Need for Stage 1 Model in Stage 3 training

[sahilvora10]

Hi,

Thank you for this repository. It is great work. I have a few questions and would like to clarify some doubts.

I was successfully able to run all three stages, but I have a question: Do we need to pass the Stage 1 model to Stage 3 during training? I noticed that when I train the Stage 3 diffusion stage by loading the network of Stage 1 results through the config file, the results are better.

I pass the Stage 1 trained model under this:
"noise_model": { "resume_state":

However, if the Stage 1 Noise model is not passed through Stage 3 training, then the results drop and the network is not able to learn even after all the iterations.

Can you explain why that might be the case and why the weights and state dictionary of Stage 1 are needed for Stage 3?

Thank you.

[tiangexiang]

Hi, thanks for your interest in our work, and sorry for the late reply.
Yes, loading Stage1 trained model for Stage3 is a necessary step. For each iteration at stage3, we require a 'clean' image to be generated by the noise model (Stage1) (see paper fig.2). Without the pretrained Stage1 model, such a 'clean' image is not possible and Stage3 will result in degenerated solutions.