How to set fcst_thresh in Ensemble-Stat tool?

[GwenChen-NOAA]

Dear METplus team,

I need to set both fcst_thresh and obs_thresh to be greater than a value in ensemble_stat. I set the following in my config file:

OBS_VAR1_OPTIONS = obs_thresh = [ >= 271.15 ];
FCST_VAR1_OPTIONS = fcst_thresh = [ >= 271.15 ];

But, in my ensemble_stat ECNT line type output, only OBS_THRESH is set. I checked the MET Users Guide for Ensemble-Stat tool. I only see obs_thresh option available in the config file. How to set fcst_thresh in Ensemble-Stat tool? Also, I need to set the logic to be "INTERSECTION", how to do this in ensemble_stat? Thanks!

[JohnHalleyGotway]

Hello @GwenChen-NOAA, I see you have questions about setting the fcst_thresh and obs_thresh settings for MET's ensemble_stat tool. When configuring the MET tools, there are many options for defining thresholds in multiple tools and contexts. And the function of each is not as intuitive and obvious as would be ideal.

Since you mention applying INTERSECTION logic, I'm not entirely sure what you're trying accomplish with these threshold settings.

Do you want to define thresholds for events for probabilities, like deriving the ensemble probability of precipitation greater than 0, or temperature being less than 273? I'm guessing not, because no "logic" option applies to that.

Or do you want to define thresholds to FILTER the data and only compute statistics on the subset of matched pairs that meet the criteria? Since "logic" does apply to data filtering, I'm assuming this is what you're trying to do.

Can you please clarify what. you're trying to do? That'll help me understand what config options would be involved.

[GwenChen-NOAA]

John, I'm trying to do the same thing Shannon did for grid_stat (METplus Discussion #2869 <#2869>). For GEFS SST verification, we need to filter out fcst/obs temperature data < 271.15, and only compute statistics on the subset of matched pairs that meet the criteria. I use the ensemble_stat tool and output the ECNT line type.

…

On Tue, Feb 11, 2025 at 11:51 AM John Halley Gotway < ***@***.***> wrote: Hello @GwenChen-NOAA <https://github.com/GwenChen-NOAA>, I see you have questions about setting the fcst_thresh and obs_thresh settings for MET's ensemble_stat tool. When configuring the MET tools, there are many options for defining thresholds in multiple tools and contexts. And the function of each is not as intuitive and obvious as would be ideal. Since you mention applying INTERSECTION logic, I'm not entirely sure what you're trying accomplish with these threshold settings. Do you want to define thresholds for events for probabilities, like deriving the ensemble probability of precipitation greater than 0, or temperature being less than 273? I'm guessing not, because no "logic" option applies to that. Or do you want to define thresholds to FILTER the data and only compute statistics on the subset of matched pairs that meet the criteria? Since "logic" does apply to data filtering, I'm assuming this is what you're trying to do. Can you please clarify what. you're trying to do? That'll help me understand what config options would be involved. — Reply to this email directly, view it on GitHub <#2894 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXF3DCOYLUIBWXBA7F6JJL2PITAHAVCNFSM6AAAAABW3BT2WCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMJUG4YTCNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[JohnHalleyGotway]

@GwenChen-NOAA, great, thanks for clarifying what you're looking for.

This is called conditional verification, where you subset the data based on their values, prior to computing statistics. This conditional verification is supported with the cnt_thresh and cnt_logic config options in the Grid-Stat, Point-Stat, and (new) Pair-Stat tools, and this excerpt from the MET User's Guide describes it:

The “cnt_thresh” entry is an array of thresholds for filtering data prior to computing continuous statistics and partial sums.
The “cnt_logic” entry may be set to UNION, INTERSECTION, or SYMDIFF and controls the logic for how the forecast and observed cnt_thresh settings are combined when filtering matched pairs of forecast and observed values.

Shannon's discussion in #2869 was in reference to the Grid-Stat tool and she used METplus to define cnt_thresh and cnt_logic for the forecast and observation data by setting the METplus FCST_VAR1_OPTIONS and OBS_VAR1_OPTIONS config options.

But you're running Ensemble-Stat which does not support cnt_thresh and cnt_logic. Instead, only obs_thresh is provided as a way to filter pairs. But that's based only on the observation values, not the forecast ones. We did not implement filtering of ensemble values because the logic is rather confusing. Do you throw out a matched pair if ANY of the ensemble values don't meet the criteria, ALL of them don't meet it, or some configurable fraction of them? Since we never had a direct request to make this configurable, we avoided that complexity.

However, there is another alternative to get the equivalent outcome. Instead of setting obs_thresh to filter the obs data... or enhancing Ensemble-Stat to fully support cnt_thresh and cnt_logic, you could use the existing data censoring options. For example, the following settings tell METplus to treat any values < 271.15 as if they were bad data:

FCST_VAR1_OPTIONS = censor_thresh = [ <271.15 ]; censor_val = [ -9999 ]; vld_thresh = 1.0;
OBS_VAR1_OPTIONS = censor_thresh = [ <271.15 ]; censor_val = [ -9999 ];

For both the forecast (ensemble) and observation data, replace any data values <271.15 with MET's internal bad data value of -9999.

Any point where the observation value is bad data will be skipped because you can't compute stats with bad obs values.
And for the ensemble member values, the vld_thresh setting defines the ratio of ensemble members that must contain valid data for that point to be used. Using 1.0, as above, indicates that any point with any ensemble member values <271.15 should be skipped since the ratio of value data would be less than 1.0. Decreasing that to 0.5 would require that only half the ensemble inputs be >=271.15.

That would have the same effect as setting cnt_thresh = <271.15; cnt_logic = INTERSECTION;. Does that seem like a workable solution? Note that you could also include desc = GE271.15 in the OBS_VAR1_OPTIONS setting to populate the description (DESC) output column to note that you applied this filtering logic.

[GwenChen-NOAA]

@JohnHalleyGotway, thank you for the detailed explanation! It helps me have a deeper understanding of the thresh options in METplus. I followed your suggestion and did a test run. The new stats in ECNT output look much reasonable and are very close to Shannon's numbers. I will keep the job running in real time to generate some plots. Many thanks!

[JohnHalleyGotway]

@GwenChen-NOAA sounds great. I'll go ahead and close this discussion. But please feel free to open a new one if more issues or questions arise.