Clarification on how metrics are calculated

[elephaint]

If I run the following:

import pandas as pd
df_timesfm = pd.read_csv("results/timesfm_2_0_500m/all_results.csv")
print(f"TimesFM MASE: {df_timesfm['eval_metrics/MASE[0.5]'].mean():.2f} \n"
      f"TimesFM CRPS: {df_timesfm['eval_metrics/mean_weighted_sum_quantile_loss'].mean():.2f}")

the output is:

TimesFM MASE: 1.71 
TimesFM CRPS: 0.25

whereas the leaderboard here states:

Can you explain / detail how the leaderboard is calculated or point me to where it is explained?

[cuthalionn]

Hi @elephaint,

The results for each model are standardized by seasonal naive results and then we take the geometric mean across datasets for each model. You can find the details in the source code for the leaderboard ](https://huggingface.co/spaces/Salesforce/GIFT-Eval/tree/main/src)

[elephaint]

Ah, thanks, completely overlooked that, scrolling down further it becomes obvious when seeing SeasonalNaive at 1.0. Thanks!

Tiny thing: I noticed that in your Naive notebook, you actually use the SeasonalNaive. So I assume that what in this repo notebook called naive is actually SeasonalNaive on the leaderboard? (because the notebook naive will produce a table with name naive , but the results are SeasonalNaive as far as I can tell.

[cuthalionn]

No worries, I am glad it clarifies the confusion!

In the notebook we actually use Naive because the predictor is set to NaivePredictor. We use seasonal naive as the fallback model. But the same notebook can be easily adapted for seasonal naive too. One would just need to create a SeasonalNaivePredictor(StatsForecastPredictor) predictor class and set the model type accordingly.

So the terms Naive and Seasonal Naive on the leaderboard and in the repository represent their respective models.