grid_stat: "Trouble reading observation file" for GHRSST_OSPO file

[malloryprow]

There was a failure in operations of EVS due to MET not being able to read the GHRSST-OSPO (SST observation file).

The log shows

ERROR : Trouble reading observation file "/lfs/f1/ops/prod/tmp/evs_global_det_ukmet_atmos_grid2grid_stats.5789664.dbqs01/grid2grid_stats/data/ghrsst_ospo/ghrsst_ospo.2025013000to2025013100.nc"

ncdump -h is able to successfully dump the file.

I tried running regrid_data_plane SHORT_ghrsst_ospo.2025013000to2025013100.nc G004 test.nc -field 'name="analysed_sst"; level="(0,*,*)";' -v 10 and got

DEBUG 1: Start regrid_data_plane by mallory.row(32288) at 2025-02-03 13:26:54Z cmd: regrid_data_plane SHORT_ghrsst_ospo.2025013000to2025013100.nc G004 test.nc -field name="analysed_sst"; level="(0,,)"; -v 10
DEBUG 10: get_env() MET_BASE to /apps/ops/para/libs/intel/19.1.3.304/met/12.0.0-rc1/share/met
DEBUG 1: Reading data file: SHORT_ghrsst_ospo.2025013000to2025013100.nc
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_None".
ERROR :
ERROR : process_data_file() -> "SHORT_ghrsst_ospo.2025013000to2025013100.nc" not a valid data file

It is hard to say what about the file that MET is having trouble reading or what about it think is not a valid data file.

[malloryprow]

metplus_2885_1.tar.gz

[malloryprow]

This is the file MET tried to run with. We adjust the valid time to be the end of the 24 hour window from what comes in originally.

[malloryprow]

metplus_2885_2.tar.gz

[malloryprow]

Original files that NCO bring in for us.

[JohnHalleyGotway]

@malloryprow, thanks for sending along the data to demonstrate. This looks to me like a CF-compliant NetCDF file, but one that does not include the global Conventions = "CF-X.Y"; attribute to identify it as such.

Based on this log message...

DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_None".

I see that you're not using the file_type configuration option to identify this file as being CF-compliant.

Running plot_data_plane without file_type errors out:

plot_data_plane SHORT_ghrsst_ospo.2025013000to2025013100.nc plot.ps 'name="analysed_sst"; level="(@20250131_00,*,*)";'
ERROR  : plot_data_plane -> file "SHORT_ghrsst_ospo.2025013000to2025013100.nc" not a valid data file

Adding file_type=NETCDF_NCCF; to explicitly specify the file type enables it to run successfully:

plot_data_plane SHORT_ghrsst_ospo.2025013000to2025013100.nc plot.ps 'name="analysed_sst"; level="(@20250131_00,*,*)"; file_type=NETCDF_NCCF;'

And produce this plot:

The best solution would be to modify these files by adding the global Conventions = "CF-1.7"; (or whatever version) attribute, but if that's not possible, then you'll need to use the file_type=NETCDF_NCCF; config option for MET.

[malloryprow]

For EnsembleStat, would OBS_ENSEMBLE_STAT_FILE_TYPE work? Gwen doesn't see if in the METplus Configuration Glossary.

[malloryprow]

@georgemccabe

[georgemccabe]

@malloryprow, that variable is not supported. We have an open issue #2570 to add support for it. In the meantime, you can use MET config overrides to set it:

ENSEMBLE_STAT_MET_CONFIG_OVERRIDES = obs = { file_type=NETCDF_NCCF; }

[malloryprow]

Thanks! I did test that and it worked! We can mark this discussion as resolved!

[JohnHalleyGotway]

FYI, I just tested in the Ensemble-Stat tool directly and found that, yes, file_type can be specified inside the obs dictionary, but only when the -point_obs option is NOT used. You can find this logic starting on line 391 of ensemble_stat.cc.

By convention, if -point_obs is used to supply point observation data, then the file type is stored as GRIB1... since we'd historically mapped point observations to GRIB1 conventions. So when -grid_obs is supplied without -point_obs, then the obs.file_type option can be used to explicitly define the input gridded observation file type. Without it, ensemble_stat, like all the other MET tools, will inspect the input files to try to determine how to read them. And I'm not stating that this is the BEST logic we could have or that it couldn't be refined in the future as needed... just clarifying what it actually is in MET-12.0.0.

I note that in the METplus Configuration Glossary, I do not see an entry for FCST_ENSEMBLE_STAT_FILE_TYPE or OBS_ENSEMBLE_STAT_FILE_TYPE. And grep-ing through the METplus codebase, I don't see references to either. It would NOT work to put it in the OBS_VAR_OPTIONS since its parsed from inside the obs dictionary but not separately for each field.

@georgemccabe can you please advise as to if/how obs.file_type could be set with the current version of METplus and whether we should add the missing entries to the next version?

[georgemccabe]

@JohnHalleyGotway , see above for answers to both.