Adding fair scoring rules

[mpm-meto]

Recent recommendations from an in-house ensemble verification workshop made the recommendation that we should move to using fair scores across all calculations. This is because we operate a very complex ensemble generation process where we only produce 3 new members every hour and then these are combined/lagged to create an 18-member ensemble over 6h. Our system VER does not compute these internally but they are easily derived in VerPy as it is a simple scaling of the score. Therefore, whilst it isn't essential for us to have MET be able to calculate them, as we can continue to use VerPy to do so, it would be beneficial for a MET-based system to provide such scores for users of the software that don't have access to something like VerPy. I will leave it up to developers to decide whether they want to add it to the core tool or only offer it in downstream options (as we do, e.g. metcalcpy).
Fair scoring rules will ensure that irrespective of how many members the ensemble has, you can compare them safely and equitably. We would also want to leverage this for the computation of HiRA metrics.
https://rmets.onlinelibrary.wiley.com/doi/full/10.1002/qj.2270

[j-opatz]

Hello,

And thank you for your question. I've gone ahead and let our resident statistician know about this question. They're currently out of the office, but will respond to you as soon as they are back and able.
Thank you for your patience during the interim.

[RogerHar]

Note MET issue #2280 is a prerequisite to allow us to compute the fair Brier and fair RPS in VerPy from PCT output, as these scores require the probabilities to be un-binned ensemble relative frequencies, and this is our most urgent requirement. As Marion says, it would be useful if these scores were available more directly from MET or METplus, but not essential for us.

As MET's IGN is the ignorance score based on a normal distribution (aka the Dawid-Sebastiani score), the fair ignorance score defined in section 3 of Siegert et al 2019 would be another possibility. It looks straightforward enough to code as a modification of IGN (it involves the digamma function $\Psi$ for half-integer arguments, for which there's a simple formula). I'm not aware it's used much, however, and we have no current requirement for it, so I wouldn't see it as a high priority.

[ericgilleland]

I agree that we should have these scores in MET (or downstream in METplus). You might get @bgbrowntollerud's opinion, though, as I believe she has more familiarity with fair scores than I have.

[bgbrowntollerud]

I am in agreement with Eric's response; including these scores is a good idea.

[TaraJensen]

Roger - after the holidays, would it be possible for you to jump in and help us with getting this into METplus. Marion suggested you might be able to help by looking at the code to determine where to add it and advise us on how to test it.

[RogerHar]

Sure Tara. You caught me minutes before I head off, I'll be back on 2 Jan. Happy holidays!

[jprestop]

Hi @RogerHar. I'm going through our Discussions to see which ones need attention. I just thought I'd follow up on this one. Perhaps it was discussed at the last meeting - I am not sure. Please let us know what you need from us if our team doesn't already know.

[RogerHar]

Hi Julie @jprestop, I think the first priority is MET issue 2280 as that's a prerequisite for calculating the fair Brier score and fair RPS. Once 2280 is available we at the Met Office could calculate the fair Brier and RPS from the PCT output line type in our VerPy software package rather than needing a 'FAIRBRIER' column to be added to PSTD, but I imagine that would be useful for other users (or as Marion said above possibly it could then be done in metcalcpy but I'm not familiar with that). MET issue 2280 is still open but appears to be allocated to 12.0.0 Beta 4 - I don't know when that's likely to be released? It has a label alert: NEED MORE DEFINITION - I don't know what that means so let me know if it needs any more explanation from me of what's wanted.

Apologies, I've not been joining the METplus Users Telecons, so I don't know if it's been discussed there.

[jprestop]

@RogerHar Thanks for this information. beta4 is scheduled for release on March 12th, however, sometimes that date changes. I'll post a comment on that issue asking if the label alert: NEED MORE DEFINITION is still needed and asking for clarification on if we need more information from you or not.