Use of climatology in ECLV line type

[hertneky]

I have a use-case that runs GenEnsProd to generate relative frequencies (e.g. wind > 5 m/s) from a 10-member ensemble, then runs pointStat to compute ECLV for an ensemble probability, e.g.:

FCST_VAR1_NAME = WIND_Z10_ENS_FREQ_ge5
FCST_VAR1_LEVELS = Z10
FCST_VAR1_THRESH = ==0.1

From the documentation, it appears that the climatology used in this computation is simply the base rate (relative frequency of event occurring aka the "sample climatology"). By using observations from the sample for the climo, the ECLV results reflect the forecast bias from this sample. Can a "true" climatology be used in this computation? For reference, we have also been referring to this Probability and Economic Value paper from ECMWF as well as the MET documentation.

[JohnHalleyGotway]

@hertneky thanks for moving this question to discussions.

I stepped through the MET source code to remind myself about the details of ECLV. Here's a link to the code in MET that computes the cost/loss values reported in the ECLV line type. And here's the reference in the code listed a few lines above that:

//  Cost/Loss Relative Value
//  Reference: Eq. 8.75b, page 380 in Wilks, 3rd Ed. 2011

And here's a link to its description in Appendix C of the User's Guide. In that description I see:

The maximum forecast value is achieved when the cost/loss ratio equals the climatological probability.

And in the code, that "climatological probability" is provided as the "double r" argument in:

   double TTContingencyTable::cost_loss(double r) const {

Switching over to stat_columns.cc, starting on line 3442:

   at.set_entry(r, c+2,  // Cost/Loss value for Base Rate
      ct.cost_loss(ct.baser()));
...
      at.set_entry(r, col, // VALUE_i
         ct.cost_loss(eclv_points[i]));

Looking at the ECLV line type table, the VALUE_BASER column is the economic value you get when the base rate is supplied as the argument to the cost_loss function.

And then the VALUE_i column in that line type contains the economic value you get for each user-specified configurable eclv_points entry defined in the config file. Here's a description from the User's Guide about setting the eclv_points config option:

The “eclv_points” entry specifies the economic cost/loss ratio points to be evaluated. For each cost/loss ratio specified, the relative value will be computed and written to the ECLV output line. This entry may either be specified as an array of numbers between 0 and 1 or as a single number. For an array, each array entry will be evaluated. For a single number, all evenly spaced points between 0 and 1 will be evaluated, where eclv_points defines the spacing. Cost/loss values are omitted for ratios of 0.0 and 1.0 since they are undefined.

After tracing this through the code, I'm confident that using an external climatology does not impact the computation of the ECLV output line. Although MET does support the definition of thresholds relative to climatology. For for example, you could define the "event" as being ">CDP75" would be greater than the 75-th percentile of climatology. That would impact the contents of the Nx2 probabilistic contingency table... and therefore the resulting cost/loss values.

I'd recommend reading through that reference in Wilks, compare it to the other reference you're reading about, and check to see whether there are differences in the logic.

I'd also refer you to @bgbrowntollerud who's the statistician that advised on the implementation of the Economic Cost/Loss Value in MET. She could help on interpreting results.