Better train #6301

kopyl · 2023-12-23T16:30:57Z

What does this PR do?

Fixes # (issue)

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Better args

Better validation

Make better paths for saving unets and checkpoints

This is needed cause `docker logs` don't show tqdm progress bar properly

Switch logging for Docker to log every step

kopyl added 16 commits December 12, 2023 23:21

Put deepspeed plugin to a separate function

c541989

Make saving unet for deepspeed

971a4ae

Allow specifying saving unet only

4fe1d82

Adjust saving unet for multi-GPU

9947bd4

Make better logging

ef2588e

Make it possible to log images on checkpointing

165cdb2

Merge pull request #1 from kopyl/better-args

f1fde45

Better args

Utilize my scheduler for validation

71e2ed0

Make it possible to control image gen with args

9d69afa

Merge pull request #2 from kopyl/better-validation

25804de

Better validation

Make better paths for saving unets and checkpoints

753ac87

Merge pull request #3 from kopyl/better-paths

43b067f

Make better paths for saving unets and checkpoints

Switch logging for Docker to log every step

7990bd6

This is needed cause `docker logs` don't show tqdm progress bar properly

Merge pull request #4 from kopyl/logging-for-docker

fd42e93

Switch logging for Docker to log every step

Make it possible to map data for over 30 minutes

29aee80

Make it possible to use a pre-computed dataset

d205a57

kopyl closed this Dec 23, 2023