PCP Combine Derive Lookback for Long Time periods

[CPKalb]

I'm writing this discussion after an email conversation with @georgemccabe. I'm running PCP-Combine for a long time period (30 years) to derive a mean and standard deviation. My original setup contained the following:

OBS_PCP_COMBINE_METHOD = DERIVE
OBS_PCP_COMBINE_STAT_LIST = mean,stdev

OBS_PCP_COMBINE_INPUT_ACCUMS = 1Y
OBS_PCP_COMBINE_INPUT_NAMES = SWVL1
OBS_PCP_COMBINE_INPUT_LEVELS = "(0,,)"

OBS_PCP_COMBINE_DERIVE_LOOKBACK = 30Y

When I run with this setup, I get an error:
ERROR: (pcp_combine_wrapper.py:553) Could not find files to build accumulation in /glade/derecho/scratch/biswas/SFS/ERA5_sfc using template e5.moda.an.sfc.128_039_swvl1.ll025sc.{valid?fmt=%Y}010100_{valid?fmt=%Y}120100.nc
10/26 05:32:28.237Z metplus.b7ac2995 (command_builder.py:264) ERROR: (pcp_combine_wrapper.py:296) pcp_combine could not generate command

I dug around a little bit to try and figure out what was going on, and discovered that the listing of files is being set to none in lines 710-711 in pcp_combine_wrapper.py. When I print out files_found, I get exactly the list of files I was trying to select. So, I think it's finding the correct files. However, when I print the total_accum, it's 604800 (which is 0.01918 years), rather than 0. If I go farther up in the file, the starting total_accum is set to:
946684800 = 30.01918 years.

It seems like the starting accumulation isn't exactly 30 years, and so maybe this is causing the wrapper to think it hasn't found all of the files for the accumulation when in reality it has? I'm not sure why this is happening or whether it may have something to do with leap days, etc.

For now, setting the OBS_PCP_COMBINE_DERIVE_LOOKBACK in seconds (OBS_PCP_COMBINE_DERIVE_LOOKBACK = 946080000S) gets around this issue and PCP-Combine runs as expected.

However, it seems like this may be an area for improvement in the PCP-Combine wrapper. One suggestion George had was to have a threshold for determining a complete set of accums based on the input accum interval instead of always using 0. This issue is for feedback and discussion on how to improve the PCP-Combine wrapper moving forward.

Christina

[JohnHalleyGotway]

We discussed this topic during the METplus Engineering meeting on 10/29/24. And we pointed out that some units of time are well-defined, such as seconds, minutes, days, and weeks. Other units of time are indeterminate, such as months, years, decades, and centuries. For these reasons, some software systems do not support the indeterminate units. But they certainly are convenient and easily understood by users.

Recommend that @georgemccabe use this situation to write up a GitHub issue about more flexibly handling these indeterminate time units. For example, @georgemccabe considered whether we could define some tolerance in determining whether or not the total accumulation period has been fully covered. And he recommended defining that tolerance as a percentage of the total accumulation period rather and an explicit hard-coded value.

Handling these types of indeterminate time units in intuitive ways becomes increasingly important the more that METplus is applied to seasonal and climate timescales.

[georgemccabe]

This problem should be resolved in PR #2764. Details can be found in issue #2762 (see comments).