Evaluation takes 5 hours for one checkpoint.

[PatZhuang]

So I trained a basic seq2seq model resulting with 15 checkpoints named ckpt.{num}.pth. I modified the NUM_PROCESSES filed to 4 and it takes about one day to train on a GTX 2080Ti graphic card (I used the headless habitat ver v0.1.5).

However, when I run the eval script, it takes about 5 hours to evaluate one checkpoint. Is this even normal?

And what is the point of evaluating all the checkpoints (and start from the earliest one) by the way?

[erikwijmans]

Evaluation for one checkpoint can take a long time when the model is bad, 5 hours is perhaps a bit higher than I would expect, but no dramatically higher.

The point of evaluating all is to be able to look at the curve that produces as it can be quite useful.