Skip to content

Commit 0361581

Browse files
Yuze-e20huan-yinyjy415
authored
Support FLUX.1 Fill Redux InsertAnything (#1611)
* support the inference of FLUX Fill, FLUX Redux, Insert Anything * feat/fix: further improve the origin pr authored by huan-yin --------- Co-authored-by: Li Xiangyue <2792578061@qq.com> Co-authored-by: Li Xiangyue <113219380+huan-yin@users.noreply.github.com> Co-authored-by: yjy415 <2471352175@qq.com>
1 parent 9c1534c commit 0361581

22 files changed

Lines changed: 827 additions & 5 deletions

File tree

‎diffsynth/configs/model_configs.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -339,6 +339,14 @@
339339
"model_class": "diffsynth.models.flux_dit.FluxDiT",
340340
"state_dict_converter": "diffsynth.utils.state_dict_converters.flux_dit.FluxDiTStateDictConverterFromDiffusers",
341341
},
342+
{
343+
# Example: ModelConfig(model_id="black-forest-labs/FLUX.1-Fill-dev", origin_file_pattern="flux1-fill-dev.safetensors")
344+
"model_hash": "f876d2a6fb247fb7da0f84fc7fb8a823",
345+
"model_name": "flux_dit",
346+
"model_class": "diffsynth.models.flux_dit.FluxDiT",
347+
"extra_kwargs": {"input_dim": 384},
348+
"state_dict_converter": "diffsynth.utils.state_dict_converters.flux_dit.FluxDiTStateDictConverter",
349+
},
342350
{
343351
# Example: ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder/model.safetensors")
344352
"model_hash": "94eefa3dac9cec93cb1ebaf1747d7b78",
@@ -481,6 +489,21 @@
481489
"model_class": "diffsynth.models.flux_ipadapter.SiglipVisionModelSO400M",
482490
"state_dict_converter": "diffsynth.utils.state_dict_converters.flux_ipadapter.SiglipStateDictConverter",
483491
},
492+
{
493+
# Example: ModelConfig(model_id="black-forest-labs/FLUX.1-Redux-dev", origin_file_pattern="image_encoder/model.safetensors")
494+
"model_hash": "6bd5343828144a2135d783519c665e2c",
495+
"model_name": "flux_redux_image_encoder",
496+
"model_class": "diffsynth.models.flux_ipadapter.SiglipVisionModelSO400M",
497+
"state_dict_converter": "diffsynth.utils.state_dict_converters.flux_ipadapter.SiglipReduxStateDictConverter",
498+
},
499+
{
500+
# Example: ModelConfig(model_id="black-forest-labs/FLUX.1-Redux-dev", origin_file_pattern="image_embedder/diffusion_pytorch_model.safetensors")
501+
# 2-layer MLP projecting SigLIP tokens (1152) to the 4096-dim FLUX text
502+
# space; keys already match FluxReduxImageEncoder, so no converter needed.
503+
"model_hash": "d8ea4618391460bc831fddaae5fb5288",
504+
"model_name": "flux_redux_image_embedder",
505+
"model_class": "diffsynth.models.flux_redux.FluxReduxImageEncoder",
506+
},
484507
{
485508
# Example: ModelConfig(model_id="stepfun-ai/Step1X-Edit", origin_file_pattern="step1x-edit-i1258.safetensors"),
486509
"model_hash": "d30fb9e02b1dbf4e509142f05cf7dd50",

‎diffsynth/models/flux_redux.py‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
import torch
2+
import torch.nn as nn
3+
import torch.nn.functional as F
4+
5+
6+
class FluxReduxImageEncoder(nn.Module):
7+
def __init__(self, redux_dim: int = 1152, txt_in_features: int = 4096):
8+
super().__init__()
9+
self.redux_up = nn.Linear(redux_dim, txt_in_features * 3)
10+
self.redux_down = nn.Linear(txt_in_features * 3, txt_in_features)
11+
12+
def forward(self, x: torch.Tensor) -> torch.Tensor:
13+
return self.redux_down(F.silu(self.redux_up(x)))

‎diffsynth/pipelines/flux_image.py‎

Lines changed: 300 additions & 4 deletions
Large diffs are not rendered by default.

‎diffsynth/utils/state_dict_converters/flux_ipadapter.py‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,5 +28,13 @@ def SiglipStateDictConverter(state_dict):
2828
new_state_dict = {}
2929
for key in state_dict:
3030
if key.startswith("vision_model."):
31-
new_state_dict[key] = state_dict[key]
31+
new_state_dict[key] = state_dict[key]
32+
return new_state_dict
33+
34+
35+
def SiglipReduxStateDictConverter(state_dict):
36+
new_state_dict = {}
37+
for key in state_dict:
38+
if key.startswith("vision_model."):
39+
new_state_dict[key[len("vision_model."):]] = state_dict[key]
3240
return new_state_dict

‎docs/en/Model_Details/FLUX.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,10 @@ graph LR;
6161
FLUX.1-Series-->black-forest-labs/FLUX.1-dev;
6262
FLUX.1-Series-->black-forest-labs/FLUX.1-Krea-dev;
6363
FLUX.1-Series-->black-forest-labs/FLUX.1-Kontext-dev;
64+
FLUX.1-Series-->black-forest-labs/FLUX.1-Fill-dev;
65+
FLUX.1-Series-->black-forest-labs/FLUX.1-Redux-dev;
66+
black-forest-labs/FLUX.1-Fill-dev-->HuanJue/Insert-Anything;
67+
black-forest-labs/FLUX.1-Redux-dev-->HuanJue/Insert-Anything;
6468
black-forest-labs/FLUX.1-dev-->FLUX.1-dev-ControlNet-Series;
6569
FLUX.1-dev-ControlNet-Series-->alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Beta;
6670
FLUX.1-dev-ControlNet-Series-->InstantX/FLUX.1-dev-Controlnet-Union-alpha;
@@ -84,6 +88,9 @@ graph LR;
8488
| [black-forest-labs/FLUX.1-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-dev) | | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-dev.py) |
8589
| [black-forest-labs/FLUX.1-Krea-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Krea-dev) | | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-Krea-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-Krea-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-Krea-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-Krea-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-Krea-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-Krea-dev.py) |
8690
| [black-forest-labs/FLUX.1-Kontext-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Kontext-dev) | `kontext_images` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-Kontext-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-Kontext-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-Kontext-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-Kontext-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-Kontext-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-Kontext-dev.py) |
91+
| [black-forest-labs/FLUX.1-Fill-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Fill-dev) | `flux_fill_image`, `flux_fill_mask` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-Fill-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-Fill-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-Fill-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-Fill-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-Fill-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-Fill-dev.py) |
92+
| [black-forest-labs/FLUX.1-Redux-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Redux-dev) | `flux_redux_image` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-Redux-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-Redux-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-Redux-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-Redux-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-Redux-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-Redux-dev.py) |
93+
| [HuanJue/Insert-Anything](https://www.modelscope.cn/models/HuanJue/Insert-Anything) | `insert_anything_source_image`, `insert_anything_source_mask`, `insert_anything_ref_image`, `insert_anything_ref_mask` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/Insert-Anything.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/Insert-Anything.py) | - | - | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/Insert-Anything.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/Insert-Anything.py) |
8794
| [alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Beta](https://www.modelscope.cn/models/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Beta) | `controlnet_inputs` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev-Controlnet-Inpainting-Beta.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev-Controlnet-Inpainting-Beta.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-dev-Controlnet-Inpainting-Beta.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-dev-Controlnet-Inpainting-Beta.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-dev-Controlnet-Inpainting-Beta.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-dev-Controlnet-Inpainting-Beta.py) |
8895
| [InstantX/FLUX.1-dev-Controlnet-Union-alpha](https://www.modelscope.cn/models/InstantX/FLUX.1-dev-Controlnet-Union-alpha) | `controlnet_inputs` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev-Controlnet-Union-alpha.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev-Controlnet-Union-alpha.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-dev-Controlnet-Union-alpha.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-dev-Controlnet-Union-alpha.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-dev-Controlnet-Union-alpha.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-dev-Controlnet-Union-alpha.py) |
8996
| [jasperai/Flux.1-dev-Controlnet-Upscaler](https://www.modelscope.cn/models/jasperai/Flux.1-dev-Controlnet-Upscaler) | `controlnet_inputs` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev-Controlnet-Upscaler.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev-Controlnet-Upscaler.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-dev-Controlnet-Upscaler.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-dev-Controlnet-Upscaler.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-dev-Controlnet-Upscaler.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-dev-Controlnet-Upscaler.py) |
@@ -142,6 +149,13 @@ Input parameters for `FluxImagePipeline` inference include:
142149
* `flex_control_strength`: Flex model control strength.
143150
* `flex_control_stop`: Flex model control stop timestep.
144151
* `nexus_gen_reference_image`: Nexus-Gen model reference image.
152+
* `flux_fill_image`: FLUX.1-Fill model image to be inpainted.
153+
* `flux_fill_mask`: FLUX.1-Fill model inpainting mask.
154+
* `flux_redux_image`: FLUX.1-Redux model reference image.
155+
* `insert_anything_source_image`: Insert-Anything model source image, i.e., the target image to be edited.
156+
* `insert_anything_source_mask`: Insert-Anything model source image mask, specifying the region to be edited.
157+
* `insert_anything_ref_image`: Insert-Anything model reference image, providing the content to be inserted.
158+
* `insert_anything_ref_mask`: Insert-Anything model reference image mask, specifying the target object in the reference image.
145159

146160
If VRAM is insufficient, please enable [VRAM Management](../Pipeline_Usage/VRAM_management.md). We provide recommended low VRAM configurations for each model in the example code, see the table in the "Model Overview" section above.
147161

0 commit comments

Comments
 (0)