Number of epochs per fold?

[paul-reiners]

How many epochs per fold are run?

[FabianIsensee]

1000

[alaurent4]

Hello, seems relevant to ask my question here if I may.
Im using the 3D full config. The 5 folds have been running for more than 2 days than and are far from 1000 epochs.
Is there a way to do some prediction even if the models are not finished yet ?
Or is it possible to reduce the number of epochs ?

[alaurent4]

Thanks for your help! I used -chk model_best , seems pretty fair to me.

[FabianIsensee]

fair? yes. Best possible? No
:-)

[alaurent4]

But if the loss is not decreasing anymore after 100 epochs on each fold, and I pick any 'best_model' after that point, it should match with the checkpoint right ?

[FabianIsensee]

trust me when I say it is best to let it run ^^. The learning rate decreases towards the end of the training and on some (!) datasets the scores improve a little bit. Training nnU-Net does not take long (~2 days on 2080ti and that is the slowest GPU. It's 16h on A100). Just let it do its thing :-)

[alaurent4]

Okay :)

[paul-reiners]

Can the five folds be run concurrently?

…

On Wed, May 5, 2021 at 1:29 PM alaurent4 ***@***.***> wrote: Hello, seems relevant to ask my question here if I may. Im using the 3D full config. The 5 folds have been running for more than 2 days than and are far from 1000 epochs. Is there a way to do some prediction even if the models are not finished yet ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#626 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALSTK3UP7NSUHIHFIYU4FDTMGFADANCNFSM44DKG7QQ> .

[FabianIsensee]

yes. Let fold0 start training (GPU utilization is high). Then start the other folds. Waiting for fold0 to start is necessary because it will unpack the training data first and unpacking should not be done by multiple trainings simultaneously