Model Comparison Infrastructure

[AFg6K7h4fhy2]

Take at least two idata objects with the same fit and forecast period & compare.

Support comparison from

• idata objects
• FluSight submissions

[damonbayer]

Two points:

1. We should think about comparing n forecasts when designing this
2. Since scoring is an important part of model comparison, and scoring is done in scoringutils in R, we need to think carefully about how to use both ArviZ and scoringutils.

[AFg6K7h4fhy2]

Can you be more specific with this?

We should think about comparing $N$ forecasts when designing this.

Not sure what is meant here: (?) $N$ models (with one forecast dataframe per model) (?) $N$ jurisdictions per forecast dataframe (one dataframe per model).

For (2) I agree completely.

[damonbayer]

Your original post mentions comparing 2 forecasts. I think whatever approach we take should be extensible to more than 2 forecasts.

[AFg6K7h4fhy2]

Your original post mentions comparing 2 forecasts. I think whatever approach we take should be extensible to more than 2 forecasts.

Ah, I see. I should have been specific myself! at least two idata objects

[AFg6K7h4fhy2]

Adding this issue to the Backlog given that multiple, non-trivial issues (e.g. #32 , #18 , #9 , #34 ) remain to be completed before the functionality outlined in this issue will begin to be integrated.