SD3 LoRA Inference..?

[OliviaOliveiira]

Hey there!
I’m sort of new (returning) back to Linux and stuff, yet I’ve tried to look up for an inference script or the way I’d be able to actually test sd3 lora made via SimpleTuner, yet seems nothing to be found. ComfyUI gives a pile of keys errors (even though I do remember there was a commit related to keys names), aand I’ve no idea where else I could give it a try, requesting backup:)

[mhirki]

SD3 LoRAs have been working just fine for me in ComfyUI. Just copy pytorch_lora_weights.safetensors under models/loras folder in ComfyUI. Then add a "Load LoRA" node to your workflow. There's an example of a full workflow here: https://civitai.com/models/517442/copax-simple-lora-sd3-workflows?modelVersionId=574997

[BJQ123456]

How can I perform inference on a custom SD3 model with a modified architecture that I trained myself? Are there any related code examples for this?Thanks

[MohamedAliRashad]

Can i add lora to huggingface SD3 pipeline ?

[xings19]

Hi, I want to know if you get the same effect when loading your lora with comfyui as when you validate with simpletuner. I also tested with comfyui, but the effect I got was different from the validate effect of simpletuner.

[pvp-by]

@mhirki Hi friend, have you personally succeeded in training LoraSD3, if yes - could you share configs (sdxl-env.sh, multidatabackend.json) and information what branch/commit you used for success.

2 people have the same error, maybe we have stumbled on a bug that depends only on configuration files. #545

[mhirki]

Yes, I have successfully trained LoRAs for SD3. Below are my sdxl-env.sh and multidatabackend.json for training a single person/character LoRA. Obviously, you should replace "sanna marin" with the something else.

If you run out of VRAM, you can try enabling export USE_GRADIENT_CHECKPOINTING=true in sdxl-env.sh but make sure you have the latest version of diffusers installed. On the other hand, if you have plenty of VRAM available, you can increase TRAIN_BATCH_SIZE.

Changes to sdxl-env.sh (based on sdxl-env.sh.example):

export STABLE_DIFFUSION_3=true
export CHECKPOINTING_STEPS=500
export CHECKPOINTING_LIMIT=3
export LEARNING_RATE=8e-4 #@param {type:"number"}
export MODEL_NAME="stabilityai/stable-diffusion-3-medium-diffusers"
export DEBUG_EXTRA_ARGS="--report_to=tensorboard"
export MAX_NUM_STEPS=2000
export VALIDATION_PROMPT="closeup of sanna marin"
export TRAIN_BATCH_SIZE=1
export GRADIENT_ACCUMULATION_STEPS=4
export LR_SCHEDULE="constant_with_warmup"
export LR_WARMUP_STEPS=$((MAX_NUM_STEPS / 10))
export USE_XFORMERS=false
export TRAINER_EXTRA_ARGS="--max_grad_norm=0.01"

multidatabackend.json:

[
  {
    "id": "sanna-marin-sd3",
    "type": "local",
    "crop": false,
    "crop_aspect": "preserve",
    "crop_style": "random",
    "resolution": 1.0,
    "minimum_image_size": 0.25,
    "maximum_image_size": 1.0,
    "target_downsample_size": 1.0,
    "resolution_type": "area",
    "cache_dir_vae": "cache/vae/sd3/sanna_marin",
    "instance_data_dir": "datasets/sanna_marin",
    "disabled": false,
    "skip_file_discovery": "",
    "caption_strategy": "instanceprompt",
    "instance_prompt": "sanna marin",
    "metadata_backend": "json"
  },
  {
    "id": "text-embeds",
    "type": "local",
    "dataset_type": "text_embeds",
    "default": true,
    "cache_dir": "cache/text/sd3/sanna_marin",
    "disabled": false,
    "write_batch_size": 128
  }
]

[pvp-by]

time ~1h test1:

export LEARNING_RATE=8e-7
export TRAINER_EXTRA_ARGS=""

time ~1.5h test2 :

export MODEL_TYPE='lora'
export STABLE_DIFFUSION_3=true
export CHECKPOINTING_STEPS=500
export CHECKPOINTING_LIMIT=5
export LEARNING_RATE=0.0001 
export MAX_NUM_STEPS=3000
export TRAIN_BATCH_SIZE=2
export GRADIENT_ACCUMULATION_STEPS=4
export LR_SCHEDULE="sine"
export LR_WARMUP_STEPS=$((MAX_NUM_STEPS / 10))
export TRAINER_EXTRA_ARGS="${TRAINER_EXTRA_ARGS} --offset_noise --noise_offset=0.25 --noise_offset_probability=0.5"
export TRAINER_EXTRA_ARGS="${TRAINER_EXTRA_ARGS} --lora_rank=16 --lora_alpha=16 --prediction_type=v_prediction --rescale_betas_zero_snr"

multidatabackend.json

[
    {
        "id": "sd3test",
        "type": "local",
        "probability": 1.0,
        "dataset_type" : "image",
        "repeats": 0,
        "crop": false,
        "resolution": 1024,
        "resolution_type": "pixel",
        "instance_root_dir": "/home/ubuntu/SimpleTuner/datasets/data",
        "instance_data_dir": "/home/ubuntu/SimpleTuner/datasets/data",
        "cache_dir_vae": "/home/ubuntu/SimpleTuner/datasets/cache_image",
        "vae_cache_clear_each_epoch": false,
        "instance_prompt": "sd3test",
        "caption_strategy": "instanceprompt",
        "skip_file_discovery": "",
        "text_embeds": "alt-text-embeds",
        "metadata_backend": "json",
        "preserve_data_backend_cache": true
    },
    {
        "id": "alt-text-embeds",
        "type": "local",
        "dataset_type": "text_embeds",
        "default": true,
        "cache_dir": "/home/ubuntu/SimpleTuner/datasets/cache_text"
  }
]

only diff options

both time size lora ~10MB (i think very small)
image generation with lora - i see "fake screenshot editor text or code fullscreen"
my dataset futuristic flower 10 images.

What direction to look for the problem, any ideas?

[mhirki]

LEARNING_RATE=8e-7 is too small to learn anything.

--noise_offset=0.25 is probably way too high. I suggest not adding fancy options to TRAINER_EXTRA_ARGS until you have a working baseline.