Automatic continuation #363
Conversation
frostedoyster
force-pushed
the
auto-continue
branch
from
cb1c66f
to
f79a36d
Compare
frostedoyster
force-pushed
the
auto-continue
branch
from
c33bcf2
to
427d3d8
Compare
There was a problem hiding this comment.
Might be a good idea. If this moves further remember to explain this well in the docs.
We aren't writing the checkpoints of the current training run in the directory where the training is started and take this file as reference. Would be a bit easier compared to searching for the latest folder...
| @@ -98,6 +98,29 @@ def _prepare_train_model_args(args: argparse.Namespace) -> None: | |||
| args.options = OmegaConf.merge(args.options, override_options) | |||
|
|
|||
|
|
|||
| def _process_continue_from(continue_from: str) -> Optional[str]: | |||
| # covers the case where `continue_from` is `auto` | |||
| if continue_from == "auto": | |||
There was a problem hiding this comment.
We should inform the user that the training is continued if the outputs directory is found or maybe error or warn if continue="auto" and now directory is found.
There was a problem hiding this comment.
Ok! It's now there
src/metatrain/cli/train.py
Outdated
| if Path("outputs/").exists(): | ||
| # take the latest day directory | ||
| dir = sorted(Path("outputs/").iterdir())[-1] | ||
| # take the latest second directory |
There was a problem hiding this comment.
| # take the latest second directory | |
| # take the latest time directory |
There was a problem hiding this comment.
Also the first directory (the "day" directory) is technically "time", I'll make it clearer
Implements "automatic" continuation as described in #362.
Questions:
Contributor (creator of pull-request) checklist
📚 Documentation preview 📚: https://metatrain--363.org.readthedocs.build/en/363/