code for finetuning 1024 model.

[SunzeY]

I follow the instruction to finetune 512px and it works well. However, When switch to 1024px-MS version, it doesn't produce reasonable result. Is it possible to share a official implementation to finetune 1024px-MS version of your work? This will be really helpful!

[SunzeY]

This really drives me crazy! I have tried anything I can. All I do is change the config of Pokemon-example:

1. change config

_base_ = ['PixArt-alpha/configs/pixart_config/PixArt_xl2_img1024_internalms.py']
...
data = dict(type='InternalDataMS'...
model = 'PixArtMS_XL_2'
...

I train the model and convert pixart model to diffuser exactly follow the same pipeline as 512px base fine-tuning pipeline. However, my result always looks like this:

However, 512px base model works well normally...
Is there anything important missing that makes Pixart-MS-1024 different from Pixart-512? I would really appreciate it if official fine-tuning demo on MS-1024 will be made available!

[SunzeY]

May be there is something wrong with learnable scale and ratio embedding? Should I keep them fixed during training?

[youngwanLEE]

@SunzeY Same situation; I'm waiting for the author's response, too.

[SunzeY]

I will email their author to try to get feedback from them.

[ruizhaocv]

I think this is caused by the wrong positional embedding. Set the lewei_scale to 2.0 in the config file and have a try!

[SunzeY]

Thank you so much, this solve my problem!

[MingzhouZhang]

In inference.py, no-ema checkpoint is loaded, while fine-tuning ema checkpoint is saved in 'state_dict_ema' key. Replace with the following code solved my problem.
missing, unexpected = model.load_state_dict(state_dict['state_dict_ema'], strict=False)

PixArt-alpha/scripts/inference.py

Line 158 in 82c8559

missing, unexpected = model.load_state_dict(state_dict['state_dict'], strict=False)

[SunzeY]

hi, I have a small question for this. if state_dict['state_dict_ema']'s weight is better. Why not their official implementation of converting pth into hugging face format load ema shadow but still use state_dict['state_dict].

[heyalexchoi]

Does your config continue from the 1024 checkpoint? I had that issue.