SeriesAnalysis output incorrect

[wuxx66]

I have used SeriesAnalysis tool to verify 2m temperature forecast with an analysis. I verified at different lead times but the output ncfile has the same output. It does not seem that the lead time has been reflected in the log file.

Could you help me identify the problem?

I have uploaded the config file, log file, NetCDF output, sample input files under "wu_data" directory.

Thank you

Xiaoxi

[JohnHalleyGotway]

Xiaoxi, I wanted to let you know that we did see this discussion and have pulled your data. I do have it on my list to follow up but haven't found the cycles to do so yet. Thanks for your patience.

[JohnHalleyGotway]

OK, I took a closer took at the data you sent. I see that you're using METplus to run Series-Analysis, comparing forecast data from NetCDF files named preprocessed_Temp_NAT.YYYYMMDD.pm.Official.nc to analysis data from NetCDF files named preprocessed_Temp_NAT.YYYYMMDD.pm.Analysis.nc.

The time series has length 30, reading 30 timesteps from the forecast file and 30 timesteps from the observation file. However there is a 1-day mismatch in those timestamps. Looking in the metplus.log.20240910060347 log file, I see that the forecast "levels" range from level="(20230302_00,*,*)"; to level="(20230331_00,*,*)"; while the observation "levels" range from level="(20230301_00,*,*)"; to level="(20230330_00,*,*)";. So the forecast times are +1 day greater than the corresponding observation times.

These offsets yield this type of warning message in the log output:

WARNING: get_series_data() -> Forecast and observation valid times do not match 20230302_000000 != 20230301_000000 for data(20230302_00,*,*) versus data(20230301_00,*,*).

So there's definitely some question as to timestamps lining up.

You also sent Series-Analysis output files. I certainly do see that series_analysis_GFE_MSAS_stats_F12_all.nc and series_analysis_GFE_MSAS_stats_F24_all.nc appear to contain the same output. And the metadata and actual RMSE values do appear to be the same.

I grep-ed the log file you sent to retrieve the commands that generated these files:
For series_analysis_GFE_MSAS_stats_F12_all.nc:

/opt/spack/opt/spack/linux-rocky8-broadwell/gcc-8.5.0/met-11.0.2-iqskdm5xi6j3akscbrin3gedvhad6thb/bin/series_analysis -fcst /g/data/wa46/user/wuxx/science_test/GFE_hourly_T/output/SeriesAnalysis/series_analysis_files_fcst_init_ALL_valid_ALL_lead_43200.txt -obs /g/data/wa46/user/wuxx/science_test/GFE_hourly_T/output/SeriesAnalysis/series_analysis_files_obs_init_ALL_valid_ALL_lead_43200.txt -out /g/data/wa46/user/wuxx/science_test/GFE_hourly_T/output/SeriesAnalysis/series_analysis_GFE_MSAS_stats_F12_all.nc -config /opt/spack/var/spack/environments/container/.spack-env/._view/5ost77efn5ahgdmn3w62uqtjdfg3xarl/METplus/parm/met_config/SeriesAnalysisConfig_wrapped -v 5

For series_analysis_GFE_MSAS_stats_F24_all.nc:

/opt/spack/opt/spack/linux-rocky8-broadwell/gcc-8.5.0/met-11.0.2-iqskdm5xi6j3akscbrin3gedvhad6thb/bin/series_analysis -fcst /g/data/wa46/user/wuxx/science_test/GFE_hourly_T/output/SeriesAnalysis/series_analysis_files_fcst_init_ALL_valid_ALL_lead_86400.txt -obs /g/data/wa46/user/wuxx/science_test/GFE_hourly_T/output/SeriesAnalysis/series_analysis_files_obs_init_ALL_valid_ALL_lead_86400.txt -out /g/data/wa46/user/wuxx/science_test/GFE_hourly_T/output/SeriesAnalysis/series_analysis_GFE_MSAS_stats_F24_all.nc -config /opt/spack/var/spack/environments/container/.spack-env/._view/5ost77efn5ahgdmn3w62uqtjdfg3xarl/METplus/parm/met_config/SeriesAnalysisConfig_wrapped -v 5

However, I do not see the actual contents of those file list files:

series_analysis_files_fcst_init_ALL_valid_ALL_lead_43200.txt
series_analysis_files_obs_init_ALL_valid_ALL_lead_43200.txt
series_analysis_files_fcst_init_ALL_valid_ALL_lead_86400.txt
series_analysis_files_obs_init_ALL_valid_ALL_lead_86400.txt

Spending more time sifting through the log messages didn't provide a clear explanation.

Here's how I would expect things to be working...
The input file you sent (preprocessed_Temp_NAT.20230326.pm.Official.nc) has an initialization (forecast_reference_time) timestamp of 20230322_070000. A 12-hour forecast in that file should have a valid time of 20230322_190000 and a 24-hour forecast should have a valid time of 20230323_070000.

However all of the valid levels being requested data(20230302_00,*,*) are for hour 00... not 07 or 19.

So I'm pretty confused about the timestamps. I also see you're running METplus v5.0.1. It's possible that the Series-Analysis wrapper has been improved in v5.1.0 and perhaps has more changes slated for v6.0.0.

With all this uncertainty in mind, I think it'd be best to ask you to send me input files for 2 additional times:
preprocessed_Temp_NAT.20230327.pm.Analysis.nc
preprocessed_Temp_NAT.20230327.pm.Official.nc
preprocessed_Temp_NAT.20230328.pm.Analysis.nc
preprocessed_Temp_NAT.20230328.pm.Official.nc

Then I could actually run Series-Analysis myself using a time series of length 3 and try to better understand what's happening.

[wuxx66]

Hi John,
Thank you for looking into the issue I have.

I just found out the forecast file "preprocessed_Temp_NAT.20230326.pm.Official.nc" is not correct. I have uploaded this file again. This file is hourly 2m temperature forecast file. The issue date is 06UTC 202230326, however the start time of the file is 08UTC 20230325, as the file contains some forecast from previous day issue. Sorry for the confusion.

I have also uploaded another 2 day forecast and analysis files.

Thanks a lot for your help

Xiaoxi

[JohnHalleyGotway]

Xiaoxi, thanks for sending along the additional data. @georgemccabe and I met this morning to step through the details. While we found the timestamps of the filenames versus the timestamps of the data to be rather confusing, we did conclude that the Series-Analysis wrapper likely has a bug in the handling of this data.

Specifically, the problem seems to be in the Series-Analysis wrapper handling input files that contain multiple output times. While it does create the list of input files properly, it does NOT handle the timestamps referenced in the FCST_FIELD_LIST and OBS_FIELD_LIST properly.

I've asked @georgemccabe to write this up as a bugfix issue in the METplus repository. This usage of Series-Analysis won't work until that bug is fixed.

Here's some additional details about the data you sent:

1. Forecast files named (e.g. preprocessed_Temp_NAT.20230326.pm.Official.nc) contain forecast data initialized at 08Z on 1 day prior to file timestamp.

• Initialization time is defined in the forecast_reference_time variable:

int64 forecast_reference_time ;
		forecast_reference_time:units = "hours since 1980-01-01T00:00:00+00:00" ;

With:

forecast_reference_time = 378944 ;

And date -ud "19800101 + 378944 hours" +%Y%m%d_%H = 20230325_08.

• Valid time is defined in the time variable as 0 to 199 hourly offsets from:

time:units = "hours since 2023-03-25 08:00:00"

2. Analysis files named (e.g. preprocessed_Temp_NAT.20230326.pm.Analysis.nc) ALSO CONTAIN FORECAST DATA... which is unexpected. That data is initialized at 07Z on 4 days prior to the file timestamp.

• Initialization time is defined in the forecast_reference_time variable:

int64 forecast_reference_time ;
		forecast_reference_time:units = "hours since 1980-01-01T00:00:00+00:00" ;

With:

forecast_reference_time = 378871 ;

And date -ud "19800101 + 378871 hours" +%Y%m%d_%H = 20230322_07.

• Valid time is defined in the time variable as 0 to 97 hourly offsets from:

time:units = "hours since 2023-03-22 07:00:00"

So even though, the observation data is named Analysis it really is just another forecast file. And that's totally fine to compare two different forecast data sources. However, exactly WHAT you actually want to compare is less obvious. We're making the ASSUMPTION that you want to compare data from the forecast preprocessed_Temp_NAT.20230326.pm.Official.nc file to the observation preprocessed_Temp_NAT.20230326.pm.Analysis.nc file. But since the analysis data source contains 98 hours of data, each valid time appears in 4 different analysis files.

@John-Sharples and Nick Loveday are currently visiting. So I'll run these details by them to see if they have any additional input.

[wuxx66]

Hi John,
Thank you so much for looking into this issue. Your reply is very thorough and detailed. Also thank you for raising a bugfix issue.

Xiaoxi

[georgemccabe]

Hi @wuxx66, here is the GitHub issue for this enhancment: #2705
We hope to get these changes into the METplus v6.0.0 release. Is this something that you will need to be added as a bugfix to the v5.1 release or is using the next official release sufficient?

[wuxx66]

Hi @georgemccabe , very glad to hear that you have got a solution for the bugfix. We would prefer to add this bugfix to the v5.1 release if possible.

Thank you very much for your help

Xiaoxi

[georgemccabe]

Hi Xiaoxi @wuxx66,

Pull request #2707 applied this bugfix to the main_v5.1 branch. Could you obtain the METplus wrappers from that branch and test that this fixes the issue you were seeing in regards to the field info being populated with time info?

Also, will you need us to create a v5.1.1 bugfix release with this change or is obtaining the update from the main_v5.1 branch acceptable?

[georgemccabe]

Hi Xiaoxi @wuxx66,

The bug fix described in #2705 is available in the METplus v6.0.0-beta6 development release. Could you please test using this version to confirm that the fix solves the issue you described in this discussion? You can comment on discussion #2342 METplus Coordinated 6.0.0 Release Acceptance Testing to inform us if your tests passed or failed. Thank you!

[wuxx66]

Thanks a lot George. We will install METplus v6.0.0-beta6 and test it. I will let you know the test results.

[georgemccabe]

Thank you. I forgot to mention that the fix is also available in the main_v5.1 branch if you need to use v5.1. We did not create a bugfix 5.1.1 release because we only do that as requested, so please let us know if you need us to create it.

[wuxx66]

Thank you George. We don't need bugfix 5.1 release. We have tested the fix in the main v5.1 and it works.