Cannot replicate the results for loca_few_shot.pth model checkpoint

[YaelGrins]

Hi,
Thank you very much for sharing your work.
I am experiencing an issue where I cannot replicate the results published while using the loca_few_shot.pth model checkpoint file.
The results I observed after running the evaluation script are as follows:
Val set MAE: 62.53 RMSE: 127.91
Test set MAE: 58.45 RMSE: 149.80
Thanks in advance for your time and assistance.

Best regards,
Yael

[djukicn]

@YaelGrins Thank you for your question. Is your issue similar to #2, i.e., did you maybe have issues with loading the pretrained model at first?

[niki-amini-naieni]

These are the results I got for FSC-147 -

From running the evaluation code with the provided checkpoints:
Val MAE: 10.22, Val RMSE 32.52, Test MAE: 10.71, Test RMSE: 56.28

From training the model using the provided training code:
Val MAE: 10.25, Val RMSE: 34.88, Test MAE: 11.43, Test RMSE: 69.84

Most results (except test RMSE) look reasonable compared to the published ones. The test RMSE from running the training code is quite high but still SOTA for class-agnostic counting.