Using Mode for Evaluating Precipitation Forecasts with Point Data

[Daisy792]

Hello,

I am currently evaluating precipitation forecast results using MET tools, primarily Mode and Ensemble Stat. My observation file consists of station observations in CSV format, specifically focusing on 72-hour precipitation data for the South China region, totaling around 20,000 data points.

To facilitate comparison, I interpolated the forecast data onto the same grid as the observation data, resulting in another CSV file with approximately 20,000 data points. Since Mode and Ensemble Stat can only evaluate gridded files, I converted both CSV files into NetCDF files using ASCII2NC. However, I encountered an issue: Mode seems unable to read the NetCDF files, possibly due to the lack of a standard grid.

Given this situation, I am wondering if it is the only solution to interpolate the observations onto the forecast grid in order to perform the evaluation. Additionally, I would like to know if embedding a Python script could provide a solution for making Mode read point observation NetCDF files.

Any advice would be appreciated!

[JohnHalleyGotway]

Hi @Daisy792, I see you have a question about running the MODE and Ensemble-Stat tools in MET. You'd like to compare a gridded precipitation forecast to a relatively dense set of point observations. You've already interpolated those point observations to the forecast grid and then ran those through the ASCII2NC tool. But MODE is not happy with those inputs.

Let me briefly review the inputs and outputs for some of the MET tools.

• MODE compares gridded forecasts to gridded observations.
• Grid-Stat compares gridded forecasts to gridded observations.
• Point-Stat compares gridded forecasts to point observations.
• Ensemble-Stat compares gridded forecasts to gridded observations (using the -grid_obs command line option) ... OR ... point observations (using the -point_obs command line option).
• ASCII2NC reformats a variety of ASCII point observation inputs into a NetCDF point observation file format. This is the same format written by the other point pre-processing tools, like PB2NC and MADIS2NC.

MODE cannot read that NetCDF point observation file format, even if the data in it occurs at grid point locations. Python embedding is certainly one option you could use to serve up that gridded data field to MODE.

However, I'd recommend using the Point2Grid tool instead. I think you'll find it very useful. The purpose of Point2Grid is to read a relatively dense set of point observations and interpolate them onto a grid. The gridded NetCDF file it creates can be read directly by MODE.

So, I'd recommend that you consider doing the following:

1. Reformat your ~20,000 point observations into an ASCII format that ASCII2NC can read.
2. Run them through ASCII2NC to put them into MET's common NetCDF point observation format.
3. Run the ASCII2NC output through Point2Grid to interpolate those ~20,000 points onto the forecast grid.
4. Run MODE to compare your gridded forecast to the gridded output from Point2Grid.

As for Ensemble-Stat, you could compare your gridded forecast to the ~20,000 raw point observations from ASCII2NC (using -point_obs <path to ascii2nc output>) or to the gridded output from Point2Grid (using -grid_obs <path to point2grid output>). Either option will work.

Hope that helps clarify your options.

[Daisy792]

Thank you very much for your prompt response! I'll try using Python embedding to provide the data field from the NetCDF point observation file format to MODE. In fact, I had previously used the Point2Grid tool to interpolate point observations onto the same grid as the forecast file, and successfully used both the MODE and Ensemble-Stat tools—sorry I forgot to mention that earlier.

I read in the MET user guide that when using tools like MODE and Ensemble-Stat, the forecast and observation files need to be regridded onto the same grid, and it’s recommended to regrid the forecast files onto the observation grid. Since I previously interpolated the point observation file onto the forecast grid, I was wondering if I could try interpolating the forecast grid onto the observation grid for evaluation.

I know that the Point-Stat tool can apparently interpolate forecast grid points onto observation points for evaluation, but I’d like to know how the evaluation results from the Point-Stat tool differ from those obtained by MODE and Ensemble-Stat?

[JohnHalleyGotway]

Since I previously interpolated the point observation file onto the forecast grid, I was wondering if I could try interpolating the forecast grid onto the observation grid for evaluation.

Since you're starting with point observations at individual lat/lon locations, it seems to me that you don't really have an observation grid to use. I do understand that you've interpolated them to a grid. But generally speaking, scientists see interpolation and regridding as introducing a source of error and uncertainty into the verification process. And typically, they're more comfortable applying that error to the forecast data than the observation data, which they'd prefer to keep as pristine as possible. That's the reasoning behind preferring to interpolate the forecast data to the observation location.

Although resolution of the data is another perspective to consider. When comparing two gridded datasets with differing resolution, I've heard a preference for regridding and evaluating on the grid with lower resolution. The argument there is that when interpolating low-res data to a high-res grid, you're making up data values in-between the original grid points. And that made up data implies more certainty in the results than you should have. Others prefer to choose a middle-ground resolution between the two and interpolate both to that common grid. I've definitely seen both approaches used.

So in practice, you're welcome to perform whatever interpolation and regridding steps you'd like. Just be aware of the choices you're making and consider what impact they may have on the results.

This is certainly one advantage to using the object-based approach provide by MODE. The resolution of the input data matters much less than the number, location, size, and shape of the objects that are resolved from the gridded data. MODE is more resilient to questions about regridding and interpolation than a traditional grid-point by grid-point analysis, like you'd do in Grid-Stat.

The Point-Stat, MODE, and Ensemble-Stat tools all apply difference evaluation methods and generally support different combinations of input types. Generally speaking, scientists have the highest confidence in individual point observations at stations and lower confidence in a gridded representation of that point data. So when possible, I'd recommend verifying against the raw station observation with Point-Stat and/or Ensemble-Stat.

[Daisy792]

Thank you very much for your patient explanation, which has given me a new understanding of interpolation and evaluation! Moving forward, I will use Point-Stat and/or Ensemble-Stat to validate the raw station observations. Your response has been very helpful to me!

Wishing you all the best!
Jiaxin

[JohnHalleyGotway]

Thanks for the followup on this discussion and selecting an "Answer". I appreciate it. And I hope you continue to find the METplus tools useful!