Optional quasirandom timesteps, zero terminal SNR, cosine schedule for SD models #138
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Landanjs
reviewed
Apr 22, 2024
There was a problem hiding this comment.
Nice! I had two questions about the quasirandom sampling, one clarification on the schedulers, and a nit.
I'm not familiar with Sobol sequences, so I don't have an intuition if this is the best way to do it, but I trust your decision! From the description of Sobol sequences it sounds like what we need
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR enables a few optional tweaks to the training process for SD style models that differ from the standard training recipe:
quasirandomness: Select timesteps quasi-randomly to reduce loss variancezero_terminal_snr: Rescale the noise schedule to include zero SNRbeta_schedule: Allows one to select fromscaled_linear(default),linear, orsquaredcos_cap_v2noise schedules.By default these are the standard values for SD2 and SDXL.