Error when running pcp_combine with IMERG data

[briannen]

Hi!

I am currently trying to run pcp_combine with IMERG data (currently working on hera). I am using python embedding with the sum_IMERG_V06_HDF5.py script in /contrib/METplus/METplus-4.1.4/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5.py. When I try to run METplus, I am getting the following error:

Warning! HDF5 library version mismatched error
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as 'LD_LIBRARY_PATH'.
You can, at your own risk, disable this warning by setting the environment
variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.
Setting it to 2 or higher will suppress the warning messages totally.
Headers are 1.10.2, library is 1.8.18
SUMMARY OF THE HDF5 CONFIGURATION
=================================

General Information:

               HDF5 Version: 1.8.18
              Configured on: Mon Mar 28 19:04:42 UTC 2022
              Configured by: role.metplus@hfe02
             Configure mode: production
                Host system: x86_64-unknown-linux-gnu
          Uname information: Linux hfe02 3.10.0-1160.53.1.el7.x86_64 #1 SMP Fri Jan 14 13:59:45 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
                   Byte sex: little-endian
                  Libraries: static, shared
         Installation point: /contrib/met/10.1.0/external_libs

Compiling Options:

           Compilation Mode: production
                 C Compiler: /apps/oneapi/compiler/2022.0.2/linux/bin/intel64/icc ( Intel(R) C Intel(R) 64 Compiler Version 2021.5.0 Build 20211109_000000)
                     CFLAGS: -fPIC
                  H5_CFLAGS: -std=c99  -O3
                  AM_CFLAGS:
                   CPPFLAGS: -I/contrib/met/10.1.0/external_libs/include
                H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200112L   -DNDEBUG -UH5_DEBUG_API
                AM_CPPFLAGS:
           Shared C Library: yes
           Static C Library: yes

Statically Linked Executables: no
LDFLAGS: -L/contrib/met/10.1.0/external_libs/lib:
H5_LDFLAGS:
AM_LDFLAGS:
Extra libraries: -lz -ldl -lm
Archiver: ar
Ranlib: ranlib
Debugged Packages:
API Tracing: no

Languages:

                    Fortran: no

                        C++: no

Features:

              Parallel HDF5: no
         High Level library: yes
               Threadsafety: no
        Default API Mapping: v18

With Deprecated Public Symbols: yes
I/O filters (external): deflate(zlib)
MPE: no
Direct VFD: no
dmalloc: no
Clear file buffers before write: yes
Using memory checker: no
Function Stack Tracing: no
Strict File Format Checks: no
Optimization Instrumentation: no
Bye...

If I am understanding the error correctly, it looks like the IMERG files were created with a newer version of HDF5 than is being used with METplus. Is that correct? If so, how can I use this data with METplus?

[georgemccabe]

Hi @briannen, the error you are seeing is a conflict between 2 versions of the HDF5 library that are used by MET. There is a specific version of the HDF5 library needed to install the MET tools that conflicts with the version of HDF5 that is used by Python to run the Python Embedding script in that use case to read the data into MET. To prevent this error, you can set the MET_PYTHON_EXE environment variable to the full path of a version of Python that has the hdf5 library available. You can set this variable in a few ways:

1. Set the variable in the terminal or a bash script before calling run_metplus.py:

MET_PYTHON_EXE=/contrib/anaconda/anaconda3/latest/bin/python3

2. Set the variable in the [user_env_vars] section of a METplus config file. In your user config file, add the following to the bottom of the file:

[user_env_vars]
MET_PYTHON_EXE = /contrib/anaconda/anaconda3/latest/bin/python3

3. Set the variable as a command line argument to the run_metplus.py script:

run_metplus.py \
    /contrib/METplus/METplus-4.1.4/parm/use_cases/met_tool_wrapper/PCPCombine/PCPCombine_python_embedding.conf \
    /path/to/user.conf \
    user_env_vars.MET_PYTHON_EXE=/contrib/anaconda/anaconda3/latest/bin/python3

I was able to get the use case example to work on Hera after setting this variable. Please let me know if this does not work for you or if you have any other issues.

[briannen]

Thank you so much @georgemccabe!