PCP-Combine Run Sum on NBM files

[fortworthjack]

I was trying to asses if I've learned ANYTHING in this course, :), and was attempting to try to alter some of the Run Sum examples shown here
https://dtcenter.org/metplus-practical-session-guide-version-4-0/session-1-metplus-setupgrid-grid/met-tool-pcp-combine/run-sum-command

but instead use a series of 3 input GRIB files containing forecast hours from a 13UTC run of the NBM, located on AWS, like the 3 grabbed from here:
wget https://noaa-nbm-grib2-pds.s3.amazonaws.com/blend.20220215/13/core/blend.t13z.core.f119.co.grib2
wget https://noaa-nbm-grib2-pds.s3.amazonaws.com/blend.20220215/13/core/blend.t13z.core.f125.co.grib2
wget https://noaa-nbm-grib2-pds.s3.amazonaws.com/blend.20220215/13/core/blend.t13z.core.f131.co.grib2

How can I learn what I need to know about the GRIB files' contents (variable names, etc.) in order to run the Sum command to add up PCP amount forecasts over the 3 times steps. I can see the options in the sample, but I'm unsure how to get those settings out of my GRIB files so I can set them.

Thinking about the above, I was also wondering if others have examples out there, of how you may have built scripts around the necessary steps to pull off something as basic as summing PCP amounts over model output time steps. I see this example, but I still need some help on cross-referencing GRIB file contents to the needed settings. https://metplus.readthedocs.io/en/main_v4.0/generated/model_applications/precipitation/MTD_fcstHRRR-TLE_FcstOnly_RevisionSeries_GRIB.html#sphx-glr-generated-model-applications-precipitation-mtd-fcsthrrr-tle-fcstonly-revisionseries-grib-py

That is, I can see all these GREAT METplus components that we've covered in the course, but finding a place (I know there are some tutorials out there) where several of these steps are combined into a nice script or notebook to house it all would be awesome, if it exists. I've explored a few on github spaces, but it often comes down to one library missing, that I then struggle to add to my environment (Win desktop), which is different/apart from the DTC-supplied AWS environment.

Lastly, as I use the DTC-supplied AWS environment, but if I'm to build more around these components, I was hoping there might be a way (areas), beyond just the command line, in which to stitch METplus (and maybe MODE, etc.) pieces together more easily, to begin to build something that can persist in an environment I can hope to maintain and keep alive through updates, etc.

Hopefully someone can steer me in some directions on how best to progress. Apologies for being so far behind in trying to articulate such a basic UseCase-oriented question.

If this is not the proper area to ask this question, feel free to move it.

[georgemccabe]

Hi @fortworthjack,

How can I learn what I need to know about the GRIB files' contents (variable names, etc.) in order to run the Sum command to add up PCP amount forecasts over the 3 times steps. I can see the options in the sample, but I'm unsure how to get those settings out of my GRIB files so I can set them.

The pcp_combine tool will read fields named APCP from GRIB data unless explicitly specified in the command using the "-field" option. Using the -sum method, the input and output accumulations values determine which precipitation accumulation fields to read from the file.

Here is the section of the MET User's Guide that describes the tool:
https://met.readthedocs.io/en/latest/Users_Guide/reformat_grid.html#pcp-combine-tool

Towards the bottom of this section of the MET User's Guide lists the different GRIB options that can be used to extract the desired fields from GRIB files:
https://met.readthedocs.io/en/develop/Users_Guide/config_options.html#fcst

I recommend using the MET plot_data_plane tool to test these settings to ensure you are reading the desired records. Once you have confirmed that the correct record is being read using the field settings, then you can be confident that those settings will allow the data to be read by the other MET tools.

I downloaded the files you are trying to process, but received an error trying to read them into MET:

ERROR : MetGrib2DataFile::data_plane() -> PDS template number (6) is not supported. Please email [email protected].

It looks like the grid of these data is not supported in the MET tools. There may be utilities available to convert these files into a format that MET can read. I created a GitHub issue to add support for reading files with this grid: dtcenter/MET#2055
Side note: The email address listed in the error should actually direct users to the METplus Discussions page and there is a GitHub issue to update this message (dtcenter/MET#2054)

Thinking about the above, I was also wondering if others have examples out there, of how you may have built scripts around the necessary steps to pull off something as basic as summing PCP amounts over model output time steps.

Here are the existing use cases in v4.0.0 that use the PCPCombine wrapper:
https://metplus.readthedocs.io/en/v4.0.0/search.html?q=PCPCombineToolUseCase&check_keywords=yes&area=default

That is, I can see all these GREAT METplus components that we've covered in the course, but finding a place (I know there are some tutorials out there) where several of these steps are combined into a nice script or notebook to house it all would be awesome, if it exists.

Most of the use cases examples in the parm/use_cases/model_applications directory of the METplus repository run more than one tool to demonstrate how to string these commands together in a useful application.

I've explored a few on github spaces, but it often comes down to one library missing, that I then struggle to add to my environment (Win desktop), which is different/apart from the DTC-supplied AWS environment.

METplus is not currently supported directly on Windows machines. However, users have had success installing the MET tools in a WSL environment or using Docker to run METplus on Windows.

Docker is likely the easiest way to get going since the MET tools are already installed inside Docker containers. See this discussion comment for information on how to get started using Docker to run METplus: #1168 (comment)

I hope that helps. Please let me know if I didn't properly address any of your questions or if you have any other questions. Since your questions have deviated from the original PCPCombine question, I would request that you create a new discussion topic with any specific questions unrelated to the title so that other users can more easily find answered to their related questions.

[jprestop]

Hi @fortworthjack. I work with @georgemccabe, who is out of the office for a bit. I just wanted to follow up on this Discussion and see if he was able to answer your questions? If not, please let us know what other questions you have. If so, could you please select his response as the answer? Then, we'll go ahead and lock this Discussion. We want to encourage users to ask new questions in new discussions rather than posting to old ones. Hopefully that'll make the questions/answers easier for other users to follow. So if/when more issues/questions arise, please feel free to start a new discussion. Thanks!

[fortworthjack]

Hi @fortworthjack,

How can I learn what I need to know about the GRIB files' contents (variable names, etc.) in order to run the Sum command to add up PCP amount forecasts over the 3 times steps. I can see the options in the sample, but I'm unsure how to get those settings out of my GRIB files so I can set them.

The pcp_combine tool will read fields named APCP from GRIB data unless explicitly specified in the command using the "-field" option. Using the -sum method, the input and output accumulations values determine which precipitation accumulation fields to read from the file.

Here is the section of the MET User's Guide that describes the tool: https://met.readthedocs.io/en/latest/Users_Guide/reformat_grid.html#pcp-combine-tool

Towards the bottom of this section of the MET User's Guide lists the different GRIB options that can be used to extract the desired fields from GRIB files: https://met.readthedocs.io/en/develop/Users_Guide/config_options.html#fcst

I recommend using the MET plot_data_plane tool to test these settings to ensure you are reading the desired records. Once you have confirmed that the correct record is being read using the field settings, then you can be confident that those settings will allow the data to be read by the other MET tools.

I downloaded the files you are trying to process, but received an error trying to read them into MET:

ERROR : MetGrib2DataFile::data_plane() -> PDS template number (6) is not supported. Please email [email protected].

It looks like the grid of these data is not supported in the MET tools. There may be utilities available to convert these files into a format that MET can read. I created a GitHub issue to add support for reading files with this grid: dtcenter/MET#2055 Side note: The email address listed in the error should actually direct users to the METplus Discussions page and there is a GitHub issue to update this message (dtcenter/MET#2054)

Thinking about the above, I was also wondering if others have examples out there, of how you may have built scripts around the necessary steps to pull off something as basic as summing PCP amounts over model output time steps.

Here are the existing use cases in v4.0.0 that use the PCPCombine wrapper: https://metplus.readthedocs.io/en/v4.0.0/search.html?q=PCPCombineToolUseCase&check_keywords=yes&area=default

That is, I can see all these GREAT METplus components that we've covered in the course, but finding a place (I know there are some tutorials out there) where several of these steps are combined into a nice script or notebook to house it all would be awesome, if it exists.

Most of the use cases examples in the parm/use_cases/model_applications directory of the METplus repository run more than one tool to demonstrate how to string these commands together in a useful application.

I've explored a few on github spaces, but it often comes down to one library missing, that I then struggle to add to my environment (Win desktop), which is different/apart from the DTC-supplied AWS environment.

METplus is not currently supported directly on Windows machines. However, users have had success installing the MET tools in a WSL environment or using Docker to run METplus on Windows.

Docker is likely the easiest way to get going since the MET tools are already installed inside Docker containers. See this discussion comment for information on how to get started using Docker to run METplus: #1168 (comment)

I hope that helps. Please let me know if I didn't properly address any of your questions or if you have any other questions. Since your questions have deviated from the original PCPCombine question, I would request that you create a new discussion topic with any specific questions unrelated to the title so that other users can more easily find answered to their related questions.

Thanks for those details George, and sorry for not being able to keep the Subject focused better.