MET throws segfaults with failed python embedding

[mollybsmith-noaa]

Hi all,

I've found that if you use python embedding in METplus, and the python script returns an error instead of its desired results (for example if one of the packages you’re trying to import does not exist in your python environment), then it can cause MET to segfault and dump large core files. Obviously the solution here is to have the python script actually succeed, but perhaps you might want to add some error handling here to avoid the segfaults?

I produced these segfaults on Jet, log file here: /home/role.amb-verif/AMDAR_PBLH_realtime_verif/verif/amdar_rt/HRRR_OPS/realtime/METplus/logs/metplus.log.20230829215619

The python script is definitely failing because the Jet admins seem to have removed netcdf4 from our conda environment. We're working on getting it back.

Thanks!
Molly

[DanielAdriaansen]

Hi @mollybsmith-noaa, thanks for logging this discussion.

I did some testing locally and reviewed the log file you sent (thank you!). The use case you are running was added as part of the METplus 5.1 coordinated release, and requires functionality that was added in MET 11.1.0 but I notice that you are using METplus 5.0.1 and MET 11.0.1. If you have access to a 5.1 coordinated release install, can you try that? I confirmed locally that I do not receive a segfault using 5.1.

Regarding the segfault, I have forwarded this info to an engineer to see if there is a more elegant way we can handle this case. I will update this discussion with any relevant issues that come out of that discussion. We've also added a note that all METplus use cases should list what version of MET/METplus they were developed with or for, and/or what version of MET/METplus is required to run the use case.

Let me know if switching to 5.1 resolves the errors you are seeing.

[mollybsmith-noaa]

Trying this now.

[mollybsmith-noaa]

Will do, those versions were only installed on Jet as of yesterday. Out of curiosity, what new features in 5.1 are required? I know that the AMDAR PBLH use case was only added in 5.1, but I thought that it was initially developed with 5.0.1?

…

On Thu, Aug 31, 2023 at 3:45 PM Dan Adriaansen ***@***.***> wrote: Hi @mollybsmith-noaa <https://github.com/mollybsmith-noaa>, thanks for logging this discussion. I did some testing locally and reviewed the log file you sent (thank you!). The use case you are running was added as part of the METplus 5.1 coordinated release, and requires functionality that was added in MET 11.1.0 but I notice that you are using METplus 5.0.1 and MET 11.0.1. If you have access to a 5.1 coordinated release install, can you try that? I confirmed locally that I do not receive a segfault using 5.1. Regarding the segfault, I have forwarded this info to an engineer to see if there is a more elegant way we can handle this case. I will update this discussion with any relevant issues that come out of that discussion. We've also added a note that all METplus use cases should list what version of MET/METplus they were developed with or for, and/or what version of MET/METplus is required to run the use case. Let me know if switching to 5.1 resolves the errors you are seeing. — Reply to this email directly, view it on GitHub <#2335 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHBWGZJA7YVTGZW4ULQYWQ3XYEAVPANCNFSM6AAAAAA4GVDAYY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[DanielAdriaansen]

Prior to MET 11.1.0 (5.1.0), there was not a convenient way to use certain tools with point observation data via Python embedding that are required to be in a specific, complex data structure. Temporary support existed in 11.0.1 (5.0.1) for this by calling a Python function that had to be imported by the user in their Python embedding script. This is what we used to develop the use case, but since the use case was being included with 5.1 we modified the use case before submitting to work with the new functionality in 5.1, which no longer requires the user to do the import and call the function themselves. MET now automatically converts the user's point observation data to whatever format is needed by the tool they are using in their workflow.

[DanielAdriaansen]

It's the fourth bullet in the menus here under "Enhancements --> Python Embedding": https://met.readthedocs.io/en/main_v11.1/Users_Guide/release-notes.html#met-version-11-1-0-release-notes-20230731

[JohnHalleyGotway]

FYI, we've also been experiencing development issues with python embedding. All of the python embedding examples are segfaulting inside the Docker image we use for testing GitHub actions. I wanted to check whether Molly's experience and the development issues were related.

But I was able to successfully run a basic example of python embedding on jet:

module load intel/2022.1.2
module load netcdf/4.7.0
module load hdf5/1.10.5
module load nco/4.9.1
module load wgrib/1.8.1.0b
module load wgrib2/2.0.8
module load R/4.0.2
module use /contrib/met/modulefiles
module load met/11.0.1
module use /contrib/met/METplus/modulefiles
module load metplus/5.0.1
plot_data_plane PYTHON_NUMPY plot.ps 'name="/contrib/met/11.0.1/MET-11.0.1/scripts/python/read_ascii_numpy.py /contrib/met/11.0.1/MET-11.0.1/data/python/fcst.txt FCST";'

That command runs without error -- whereas it fails in our development GHA Docker environment.
I'll also note that adding a bad module to load errors out nicely with the following error message:

Traceback (most recent call last):
  File "/misc/whome/John.H.Gotway/read_ascii_numpy.py", line 4, in <module>
    import bad_package
ModuleNotFoundError: No module named 'bad_package'
WARNING: 
WARNING: python_dataplane() -> an error occurred importing module "read_ascii_numpy"
WARNING: 
ERROR  : 
ERROR  : plot_data_plane -> trouble getting field "name="read_ascii_numpy.py fcst.txt FCST";" from file "PYTHON_NUMPY"
ERROR  : 

So we are already catching some bad behavior... but apparently not all of it.

[mollybsmith-noaa]

That's interesting about the Docker container segfaults. I wonder if something similar is going on in my case, since I'm submitting the verification jobs through rocoto to run on the Jet project nodes. It's hard to be 100% certain of the environment there.

[mollybsmith-noaa]

@DanielAdriaansen @JohnHalleyGotway Running with the newest versions of MET and METplus did fix the segfault! Thank you for suggesting that. They weren't available on Jet until yesterday, but I had thought that this use case was developed with the prior versions. Thanks again for the help!