Looping Through Levels

[CPKalb]

Hi:

I am running PCP-Combine on a 30 year dataset. The data is stored as 1 file per year, and each file has 12 timesteps (one per month). What I need to do is calculate a mean and standard deviation for each month across the entire dataset. I tested my setup using the following:

VALID_BEG = 2020010100
VALID_END = 2020120100
VALID_INCREMENT = 1Y

OBS_PCP_COMBINE_RUN = True

OBS_PCP_COMBINE_METHOD = DERIVE
OBS_PCP_COMBINE_STAT_LIST = mean,stdev

OBS_PCP_COMBINE_INPUT_ACCUMS = 1Y

OBS_PCP_COMBINE_INPUT_NAMES = SWVL1
OBS_PCP_COMBINE_INPUT_LEVELS = "(0,,)"

OBS_PCP_COMBINE_DERIVE_LOOKBACK = 946080000S

Note that using OBS_PCP_COMBINE_DERIVE_LOOKBACK = 30Y doesn't work (for reasons I don't want to get into here).

The above settings work fine for January, but I need to do all months, which means looping through the levels in a file. I'm struggling to figure out a good way to do this. I first tried a custom loop list using the settings below:

CUSTOM_LOOP_LIST = 0, 1, etc...
OBS_PCP_COMBINE_INPUT_LEVELS = "({custom?fmt=%s},,)"

However, this doesn't work. I get an error regarding string substitution:
File "/glade/u/home/kalb/SFS/METplus/metplus/util/string_template_substitution.py", line 356, in do_string_sub
match_result = find_and_replace_tags_in_template(match_list,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/glade/u/home/kalb/SFS/METplus/metplus/util/string_template_substitution.py", line 398, in find_and_replace_tags_in_template
raise TypeError("The key " + split_string[0] +
TypeError: The key custom was not passed to do_string_sub for template: 'name="SWVL1"; level="({custom?fmt=%s},,)";': {'valid': datetime.datetime(2020, 1, 1, 0, 0), 'lead': 0}

I know that time templates can be placed inside the level as OBS_PCP_COMBINE_INPUT_LEVELS ="({valid?fmt=%Y%m%d_%H%M%S},,)". However, each file contains a different year of data and therefore a different date. So this does not work either.

At this point, the only way I could think to do this would be to write a python embedding script that takes the current month of data as input (rather than reading the data directly). From there, I could parse the current month inside the python embedding script. If I switch the valid increment (VALID_INCREMENT = 1m), this should allow looping over all months of data.

But, before I went through the process of writing a python embedding script, I thought I would ask.... Do we have any other ways of looping through levels that I may be unaware of?

Tina

[georgemccabe]

I was going to suggest that you do something like this:

VALID_BEG = 2020010100
VALID_END = 2020120100
VALID_INCREMENT = 1m
...
OBS_PCP_COMBINE_INPUT_LEVELS = "({valid?fmt=%m?shift=-1m},*,*)"

but I don't think that you would be able to shift the first month to be 0. You could try it but I'm guessing it will not work.

It could be nice to support arithmetic operations here, e.g. ({valid?fmt=%m}-1,*,*), but that may introduce some unintended consequences. Issue #958 involves adding support for arithmetic in the begin_end_incr syntax, so that issue could be expanded to support arithmetic elsewhere.

The custom loop list approach should work, but it appears that it does not. I think I see in the code where the custom value should be passed to the do_string_sub call but is not. I could create a branch for you to test that my change fixes the issue. I'll reply here with a branch name for you to test.

@JohnHalleyGotway, can you think of any other clever ways to get the desired NetCDF data that I'm not thinking of?

[georgemccabe]

@CPKalb, I pushed a change to the branch feature_2752_pcp_custom. Please let me know if that removes the error trying to reference the custom tag in the input levels variable.

[CPKalb]

@georgemccabe, I first tried using the shift with the month as OBS_PCP_COMBINE_INPUT_LEVELS = "({valid?fmt=%m?shift=-1m},,)". As you expected, it didn't work. Shifting month one results in month 12, rather than 0 (as it should).

I then checked out the feature_2752_pcp_custom branch and ran the code with the custom tag in the levels variable. This worked as expected and I was able to loop through all the levels and therefore all months!

Tina

[georgemccabe]

That's great to hear! I can make that change available in develop.

I also realized that the valid time used here is based on the search time used to build the accumulation, so you should be able to reference the full valid time and it should return the times for each year. Did you try this?
OBS_PCP_COMBINE_INPUT_LEVELS ="({valid?fmt=%Y%m%d_%H%M%S},*,*)"

[CPKalb]

@georgemccabe, I did try OBS_PCP_COMBINE_INPUT_LEVELS ="({valid?fmt=%Y%m%d_%H%M%S},*,*)"", but it didn't work. I got a number of notifications regarding not being able to find data. I'll post the errors below

Tina

[CPKalb]

The errors I get look like this:

WARNING:
WARNING: MetNcCFDataFile::convert_time_to_offset() -> 1577836800 does not exist at time variable
WARNING:
WARNING:
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) -> the requested time 20200101_000000 for "SWVL1" variable does not exist (20190101_000000 and 20191201_000000).
WARNING:
ERROR :
ERROR : get_field() -> can't get data plane from file "/glade/derecho/scratch/biswas/SFS/ERA5_sfc/e5.moda.an.sfc.128_039_swvl1.ll025sc.2019010100_2019120100.nc"
ERROR :

I'm doing this processing on derecho if that's helpful. The .conf file is /glade/u/home/kalb/SFS/PcpCombine_obsERA5_obsOnly_soilMoisture_timetemplates.conf

Tina

[georgemccabe]

@CPKalb and I discussed this via Slack and discovered that a bug in PCPCombine derive method was preventing the use of the full valid time in the field info. I created issue #2762 and PR #2764 to fix this for 6.0.0.