Running METplus on Orion

[briannen]

I am currently trying to run METplus 5.0.1 and MET 11.0.1 on Orion using the SRW workflow. METplus is finding all the data correctly, but is failing when it is trying to actually run the MET tool. I'm getting the following error in my log file:

/apps/contrib/MET/11.0.1/bin/grid_stat: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory

I also attempted to run with METplus 5.0.0 and MET 11.0.0, which was working previously, but it is failing with the same error.

Any ideas?

[jprestop]

HI @briannen. Thank you for letting us know about this problem. Looking at the Existing Builds page, I am using the following to load METplus on Orion:

module use /apps/contrib/modulefiles
module load metplus/5.0.1

When I run grid_stat, I do not get an error:

(base) Orion-devel-1[3] role-ovp$ grid_stat
DEBUG 1: Start grid_stat by role-ovp(9175) at 2023-04-19 18:47:07Z  cmd: grid_stat 
DEBUG 2: OMP_NUM_THREADS is not set. Defaulting to 1 thread. Recommend setting OMP_NUM_THREADS for faster runtimes.
DEBUG 2: OpenMP running on 1 thread(s).

*** Model Evaluation Tools (METV11.0.1) ***

Usage: grid_stat
	fcst_file
	obs_file
	config_file
	[-outdir path]
	[-log file]
	[-v level]
	[-compress level]

	where	"fcst_file" is a gridded forecast file containing the field(s) to be verified (required).
		"obs_file" is a gridded observation file containing the verifying field(s) (required).
		"config_file" is a GridStatConfig file containing the desired configuration settings (required).
		"-outdir path" overrides the default output directory (.) (optional).
		"-log file" outputs log messages to the specified file (optional).
		"-v level" overrides the default level of logging (2) (optional).
		"-compress level" overrides the compression level of NetCDF variable (0) (optional).

Can you please tell me more information about your environment and how you are loading METplus?

[jprestop]

@briannen Thank you. Could you please run the following and send the output?

echo $LD_LIBRARY_PATH
echo $PYTHONPATH

[briannen]

Orion-login-4[10] bnelson$ echo $LD_LIBRARY_PATH
/apps/intel-2020.2/intel-2020.2/compilers_and_libraries_2020.2.254/linux/compiler/lib/intel64_lin:/apps/intel-2020.2/intel-2020.2/compilers_and_libraries_2020.2.254/linux/ipp/lib/intel64:/apps/intel-2020.2/intel-2020.2/compilers_and_libraries_2020.2.254/linux/mkl/lib/intel64_lin:/apps/intel-2020.2/intel-2020.2/compilers_and_libraries_2020.2.254/linux/tbb/lib/intel64/gcc4.8:/apps/intel-2020.2/intel-2020.2/debugger_2020/python/intel64/lib:/apps/intel-2020.2/intel-2020.2/debugger_2020/libipt/intel64/lib:/apps/intel-2020.2/intel-2020.2/compilers_and_libraries_2020.2.254/linux/daal/lib/intel64_lin:/apps/intel-2020.2/intel-2020.2/compilers_and_libraries_2020.2.254/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/apps/intel-2020.2/intel-2020.2/compilers_and_libraries_2020.2.254/linux/daal/../tbb/lib/intel64_lin/gcc4.8:/lib64:/usr/lib64:/lib:/usr/lib:/usr/lib64/qt-3.3/lib:/opt/slurm/lib:/opt/slurm/lib/slurm
Orion-login-4[11] bnelson$ echo $PYTHONPATH

(There is nothing for $PYTHONPATH)

[jprestop]

Thank you @briannen. Your outputs are exactly the same as mine. I modified the permissions of the libpython3.8.so.1.0 file. Could you please get a clean shell, load the modules and run grid_stat to see if you get the same error? If you do, please run the following:

export LD_LIBRARY_PATH=/work/noaa/ovp/miniconda/miniconda3/envs/metplus_v5.0_py3.8/lib:${PATH}

and try again. Please let me know if one of these end up working. You shouldn't need to add /work/noaa/ovp/miniconda/miniconda3/envs/metplus_v5.0_py3.8/lib to your LD_LIBRARY_PATH, but I do expect that to fix the problem you are experiencing.

[briannen]

Orion-login-2[1] bnelson$ module load contrib
Orion-login-2[2] bnelson$ module load intel/2020.2
Orion-login-2[3] bnelson$ module load met/11.0.1
Orion-login-2[4] bnelson$ /apps/contrib/MET/11.0.1/bin/grid_stat
/apps/contrib/MET/11.0.1/bin/grid_stat: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory
Orion-login-2[5] bnelson$ export LD_LIBRARY_PATH=/work/noaa/ovp/miniconda/miniconda3/envs/metplus_v5.0_py3.8/lib:${PATH}
Orion-login-2[6] bnelson$ module list

Currently Loaded Modules:

1. contrib/0.1 2) intel/2020.2 3) met/11.0.1

Orion-login-2[7] bnelson$ /apps/contrib/MET/11.0.1/bin/grid_stat
/apps/contrib/MET/11.0.1/bin/grid_stat: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory
Orion-login-2[8] bnelson$ echo $LD_LIBRARY_PATH
/work/noaa/ovp/miniconda/miniconda3/envs/metplus_v5.0_py3.8/lib:/apps/contrib/MET/11.0.1/bin:/apps/contrib/MET/11.0.1/external_libs/bin:/work/noaa/ovp/miniconda/miniconda3/envs/metplus_v5.0_py3.8/bin:/apps/intel-2020.2/intel-2020.2/compilers_and_libraries_2020.2.254/linux/bin/intel64:/apps/intel-2020.2/intel-2020.2/compilers_and_libraries_2020.2.254/linux/bin:/apps/intel-2020.2/intel-2020.2/debugger_2020/gdb/intel64/bin:/sbin:/usr/sbin:/bin:/usr/bin:/usr/lib64/qt-3.3/bin:/apps/sbin:/apps/bin:/opt/slurm/bin
Orion-login-2[9] bnelson$ echo $PATH
/apps/contrib/MET/11.0.1/bin:/apps/contrib/MET/11.0.1/external_libs/bin:/work/noaa/ovp/miniconda/miniconda3/envs/metplus_v5.0_py3.8/bin:/apps/intel-2020.2/intel-2020.2/compilers_and_libraries_2020.2.254/linux/bin/intel64:/apps/intel-2020.2/intel-2020.2/compilers_and_libraries_2020.2.254/linux/bin:/apps/intel-2020.2/intel-2020.2/debugger_2020/gdb/intel64/bin:/sbin:/usr/sbin:/bin:/usr/bin:/usr/lib64/qt-3.3/bin:/apps/sbin:/apps/bin:/opt/slurm/bin

I'm still getting the same error.

[jprestop]

@briannen Ugh. I see I accidentally typed the incorrect command for you to run. Could you please get a clean shell and set LD_LIBRARY_PATH in the following way (I accidentally had you add PATH to the end instead of LD_LIBRARY_PATH initially):

export LD_LIBRARY_PATH=/work/noaa/ovp/miniconda/miniconda3/envs/metplus_v5.0_py3.8/lib:${LD_LIBRARY_PATH}

then load the modules and run grid_stat to see if you get the same error?

[briannen]

Yes, same error :(

Orion-login-1[1] bnelson$ export LD_LIBRARY_PATH=/work/noaa/ovp/miniconda/miniconda3/envs/metplus_v5.0_py3.8/lib:${LD_LIBRARY_PATH}
Orion-login-1[2] bnelson$ module list
No modules loaded
Orion-login-1[3] bnelson$ module load contrib
Orion-login-1[4] bnelson$ module load intel/2020.2
Orion-login-1[5] bnelson$ module load met/11.0.1
Orion-login-1[6] bnelson$ module list

Currently Loaded Modules:

1. contrib/0.1 2) intel/2020.2 3) met/11.0.1

Orion-login-1[7] bnelson$ /apps/contrib/MET/11.0.1/bin/grid_stat
/apps/contrib/MET/11.0.1/bin/grid_stat: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory

[jprestop]

@briannen Thank you. Unfortunately, I am very surprised and perplexed by this behavior. Could you please run the following and attach the output file printenv_out.txt here?

printenv >& printenv_out.txt

[briannen]

printenv_out.txt

[jprestop]

@briannen Your settings look the same as mine. Let's plan for a screen sharing session. I'll set one up and then we can report back here, after we have, hopefully, found the problem and a solution.

[jprestop]

A screen sharing session revealed that the directory /work/noaa/ovp/miniconda/ did not have the correct permissions. Permissions were only open to owner and group, but not other. Running "chmod -R 755 miniconda" resolved the problem. Thank you @JohnHalleyGotway and @briannen for meeting and thank you @TaraJensen for suggesting we meet.