How to use sd_embed to fix Token indices sequence length is longer than the specified maximum sequence length for this model (95 > 77)

[nitinmukesh]

@asomoza
Please could you suggest

Working code with token error and log without sd_embed

import torch
from diffusers import FluxPipeline, FluxTransformer2DModel
from diffusers import GGUFQuantizationConfig
from diffusers import BitsAndBytesConfig
from transformers import T5EncoderModel
from sd_embed.embedding_funcs import get_weighted_text_embeddings_flux1

bfl_repo = "black-forest-labs/FLUX.1-dev"
dtype = torch.bfloat16
prompt="Photograph capturing a woman seated in a car, looking straight ahead. Her face is partially obscured, making her expression hard to read, adding an air of mystery. Natural light filters through the car window, casting subtle reflections and shadows on her face and the interior. The colors are muted yet realistic, with a slight grain that evokes a 1970s film quality. The scene feels intimate and contemplative, capturing a quiet, introspective moment, mj"
pipe = FluxPipeline.from_pretrained(
    bfl_repo,
    torch_dtype=dtype,
)
pipe.enable_model_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()
image = pipe(
    prompt, 
    num_inference_steps=4, 
    guidance_scale=0.0
).images[0]

image.save("test.png")

Loading checkpoint shards: 100%|████████████████████████████████████| 2/2 [00:00<00:00,  9.69it/s]
Loading pipeline components...:  71%|██████████████████████▏        | 5/7 [00:00<00:00, 11.10it/s]You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers
Loading pipeline components...: 100%|███████████████████████████████| 7/7 [00:01<00:00,  6.44it/s]
Token indices sequence length is longer than the specified maximum sequence length for this model (95 > 77). Running this sequence through the model will result in indexing errors
The following part of your input was truncated because CLIP can only handle sequences up to 77 tokens: ['. the scene feels intimate and contemplative, capturing a quiet, introspective moment, mj']
  0%|                                                                       | 0/4 [00:56<?, ?it/s]

Not working code with sd_embed / same token error

import torch
from diffusers import FluxPipeline, FluxTransformer2DModel
from diffusers import GGUFQuantizationConfig
from diffusers import BitsAndBytesConfig
from transformers import T5EncoderModel
from sd_embed.embedding_funcs import get_weighted_text_embeddings_flux1

bfl_repo = "black-forest-labs/FLUX.1-dev"
dtype = torch.bfloat16
prompt="Photograph capturing a woman seated in a car, looking straight ahead. Her face is partially obscured, making her expression hard to read, adding an air of mystery. Natural light filters through the car window, casting subtle reflections and shadows on her face and the interior. The colors are muted yet realistic, with a slight grain that evokes a 1970s film quality. The scene feels intimate and contemplative, capturing a quiet, introspective moment, mj"
print("prompt encoding begin...")
# First do text encoder tasks

quant_config = BitsAndBytesConfig (
    load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16
)

text_encoder_2 = T5EncoderModel.from_pretrained(
    bfl_repo,
    subfolder="text_encoder_2",
    quantization_config=quant_config,
    torch_dtype=dtype,
)
pipe = FluxPipeline.from_pretrained(
    bfl_repo,
    transformer=None,
    vae=None,
    text_encoder_2=text_encoder_2,
    torch_dtype=dtype,
)
pipe.enable_model_cpu_offload()
prompt_embeds, pooled_prompt_embeds = get_weighted_text_embeddings_flux1(
    pipe        = pipe
    , prompt    = prompt
)
print("prompt encoding ends...")
print("clear cache...")
pipe.remove_all_hooks()
del pipe
torch.cuda.empty_cache()
print("cache cleared...")

pipe = FluxPipeline.from_pretrained(
    bfl_repo,
    text_encoder=None,
    text_encoder_2=None,
    torch_dtype=dtype,
)
pipe.enable_model_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()
image = pipe(
    prompt_embeds=prompt_embeds,
    pooled_prompt_embeds=pooled_prompt_embeds,
    num_inference_steps=4, 
    guidance_scale=0.0
).images[0]

image.save("test.png")

prompt encoding begin...
`low_cpu_mem_usage` was None, now default to True since model is quantized.
Downloading shards: 100%|███████████████████████████████████████████████████| 2/2 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|████████████████████████████████████| 2/2 [00:14<00:00,  7.09s/it]
Loading pipeline components...:  20%|██████▏                        | 1/5 [00:00<00:00,  6.34it/s]You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers
Loading pipeline components...: 100%|███████████████████████████████| 5/5 [00:00<00:00,  9.58it/s]
Token indices sequence length is longer than the specified maximum sequence length for this model (95 > 77). Running this sequence through the model will result in indexing errors
prompt encoding ends...
clear cache...
cache cleared...
Loading pipeline components...: 100%|███████████████████████████████| 5/5 [00:00<00:00,  5.05it/s]
  0%|                                                                       | 0/4 [00:23<?, ?it/s]

[asomoza]

sadly I can't help you because of a number of reasons, but the most important one is that I'm not the author or involved with that library, I did test it with SDXL and it worked without problems.

I didn't test it with Flux because it came at a later date, I almost never use Flux except for tests and I'm not a fan of using long prompts with the clip models.

I see you opened an issue with the author, and probably you will get a better solution there than here since we only can help with diffusers and the correct use of the models, this means, we can't help you with using more tokens than what the model supports (because it's hacky and difficult to test) or with external libraries, this is not because we don't want to but because we really don't have the spare time to do it.

P.S.: The warning you're getting is not one from diffusers and it's not the same error, my guess is that for the last batch of tokens the library is not correctly concatenating the tokens.

[nitinmukesh]

Thank for your input. I thought you used it with Flux based on last discussion hence I asked here.
Will wait for the author to suggest what is missing in my code.

[asomoza]

no problem, please do tell me if you can't find a solution or if it doesn't work definitely with Flux since I've been recommending it for this use case.

cc @xhinker just in case

[xhinker]

Thank you @asomoza for looping me. @nitinmukesh you can safely ignore the warning "Token indices sequence length is longer than the specified maximum sequence length for this model". it is spit out from tokenizer, actually all your tokens will be used in the embedding process

[nitinmukesh]

Thank you @xhinker for clarifying.