Can not reproduce VAE R-FVD metrics on the SkyTimelapse dataset

[BPAWD]

Thanks for open-sourcing the great work. However, I tried training the VAE on SkyTimelapse dataset for 150K steps but the R-FVD only get 66.79 while the reported number in the paper is 36.52. The PSNR results (32.62) is comparable as reported though. I also evalulate the provided pre-trained weight but only get 41.81 on R-FVD metrics on the test set and 13.47 on the training set, which are worse than the paper reported ( 36.52 on the test set and 7.37 on the training set). I use the test_ifvd() funtion in evals/eval.py for inference. Any suggestion on that?

[sihyun-yu]

Hi, thanks for your interest. FVD is greatly influenced by the number of videos used for videos --- did you use 2,048 samples for evaluation? For training, you might need to train the model longer and need to fine-tune it with adversarial loss.

[BPAWD]

Thanks for your prompt reply. I use 196 samples in the test set as shown in Table 4 in the paper to evaluate the released pre-trained checkpoints. For the UCF101 dataset, I also evaluated the released pre-trained checkpoint on 3783 samples in the testlist01.txt but only got [FVD 54.556553]. Could you release the evaluation code to facilitate the following works built upon PVDM for comparison? For the adversarial loss do you mean the d_loss in trainer.py? Do the hyperparameters need changes like using a smaller learning rate during training?

[sihyun-yu]

Hi, the results seem quite strange because I got the results similar to the original paper using the current evals.py. Did you save the video file (as mp4 format) and load it for evaluation? Due to the compression artifact it might affect the value a lot. For adversarial loss, yes - please refer to FAQ in the README file.