Question about the Gilbert Skill Score

[johnlwagner]

Greetings
I have a question about how the Gilbert Skill Score is computed for perfect forecasts. For example, here is a CTC line from one of my wind speed forecasts:

V10.0.1 NDFD NA 840000 20240401_000000 20240401_000000 000000 20240401_000000 20240401_000000 ws m/s Z10 ws m/s Z10 URMA conus NEAREST 1 >=0.0 >=0.0 NA NA CTC 1326187 1326187 0 0 0

The contingency table is:
n11=1,326,187
n10=0
n01=0
n00=0

Here is the accompanying CTS line:
V10.0.1 NDFD NA 840000 20240401_000000 20240401_000000 000000 20240401_000000 20240401_000000 ws m/s Z10 ws m/s Z10 URMA conus NEAREST 1 >=0.0 >=0.0 NA 0.05 CTS 1326187 1 1 1 NA NA 1 1 1 NA NA 1 1 1 NA NA 1 NA NA 1 1 1 NA NA NA NA NA NA NA NA NA NA NA NA 0 0 2.8966e-06 NA NA 1 1 1 NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA

Scores like CSI, POD, and FAR are correctly set to either 0 or 1. The GSS is set to NA. I believe the GSS is still computable when n10=n01=n00=0 and should be GSS=1.

C=(1326187 x 1326187)/1326187 = 2
GSS = (1326187 - 2)/(1326187 - 2) = 1

Is there an issue in computing the GSS for perfect forecasts? Is there something else in the score that I am not taking into account?
Please let me know if I am interpreting this score wrong.
Thanks
John

[JohnHalleyGotway]

@johnlwagner this is a great question. I used some sample CTC lines (ctc.txt) to test the following call to Stat-Analysis:

bin/stat_analysis -lookin ctc.txt -job aggregate_stat -line_type CTC -out_line_type CTS -out_stat cts.txt -by FCST_THRESH

And I do see the same result you report in the output. GSS = NA when the CTC line is perfect with all hits or perfect with all correct-negatives. If you remove the -by FCST_THRESH option, you'll see that when the all-hits and all-correct-negative data is combined, you do get a perfect score of 1.0.

I double-checked the GSS Equation in Appendix C and reviewed the MET code in TTContingencyTable::gss(). I do see numerically why it's being reported as bad data for both.

Plugging those numbers into the equations from Appendix C:

• n11 = T (i.e. Total) = 155 and n10 = n01 = 0.
• C = (155 + 0)*(155 + 0) / 155 = 155.
• GSS = (155 - 155) / (155 + 0 + 0 - 155) = 0 / 0 = divide-by-zero
• MET reports a bad data value of NA for divide-by-zero cases.

I cross-checked this information against the description of GSS provided by https://www.cawcr.gov.au/projects/verification/, and they look to be consistent.

While it may be technically true that 1.0 would be a perfect score, it would seem that numerically, a perfect forecast of all hits or all correct negatives yields a value of NA rather than a value of 1! Although a perfect forecast with hits > 0 AND correct negatives > 0 does indeed yield a perfect score of 1.0.

Of course the most interesting questions usually arise in these edge cases!

@ericgilleland or @bgbrowntollerud can you please weigh in here? Should the MET code be updated to report a perfect GSS score of 1.0 when the contingency table contains all hits or all correct negatives. Or should we leave the equation as-is and continue reporting NA in this divide-by-zero situation?

[johnlwagner]

Thanks for correcting my math. When computing C I added the numerator instead of multiplying.

[JohnHalleyGotway]

Ah, OK, glad you got it figured out. I'll just select and answer and close this discussion.

[johnlwagner]

NA does seem to be the correct answer when all data fall into the correct hits bin. I think 1 is more useful, but NA is technically correct.

[johnlwagner]

The Gilbert Skill Score is undefined when all data fall into the correct hits bin. The score is correctly reported as NA due to a division by zero error.