BUFRLIB compilation with oneAPI compilers?

[jprestop]

I am a software engineer on the METplus team. I am working on getting our MET code to compile on the Derecho supercomputer using the new Intel LLVM-based compilers Intel® oneAPI C++/Fortran Compiler (ICX/IFX). I am having trouble compiling bufr_v11.5.0.tar.gz. I first tried using 11.7.0, but also had problems. Then, I switched to using 11.5.0 because that is the default version being used on WCOSS2.

Have you compiled bufr_v11.5.0.tar.gz using the new intel compilers yet? Can you help me resolve these problems? Please see the attached bufr.make.log file from Derecho.
bufr.make.log

[jprestop]

Taking a closer look at the log file:

[ 7%] Building C object src/CMakeFiles/bufr_8_c.dir/icvidx.c.o
curl: (22) The requested URL returned error: 404
[ 7%] Building C object src/CMakeFiles/bufr_d_c.dir/ccbfl.c.o
make[2]: *** [test/CMakeFiles/get_bufr_test_data.dir/build.make:73: test/bufr.tar] Error 22
make[1]: *** [CMakeFiles/Makefile2:934: test/CMakeFiles/get_bufr_test_data.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

It looks like curl had some difficulty. Running grep to find where curl was referenced I saw:

jpresto@derecho6$ grep curl *
grep: bin: Is a directory
bufr.make.log:[  6%] (curl) downloading https://ftp.emc.ncep.noaa.gov/static_files/public/bufr.tar
bufr.make.log:curl: (22) The requested URL returned error: 404
CMakeCache.txt:CURL_pgm:FILEPATH=/glade/u/apps/derecho/23.06/opt/bin/curl

That led me to search for the URL in the base package:

find . -name "*" | xargs egrep -n https://ftp.emc.ncep.noaa.gov/static_files/public/bufr.tar | more`

which led me to:

./test/CMakeLists.txt:1:# Fetch test data from: https://ftp.emc.ncep.noaa.gov/static_files/public/bufr.tar

Looking in ./test/CMakeLists.txt, I see:

# Fetch test data from: https://ftp.emc.ncep.noaa.gov/static_files/public/bufr.tar
set(BUFR_URL "https://ftp.emc.ncep.noaa.gov/static_files/public")
set(BUFR_TAR "bufr.tar")

and looking at https://ftp.emc.ncep.noaa.gov/static_files/public, I can see that bufr.tar does not exist:

I had previously attempted to compile version 11.7.0 of BUFRLIB, but ran into difficult at a later step. However, I should note that test/CMakeLists.txt for version 11.7.0 still points to 11.6.0:

# Fetch test data from: https://ftp.emc.ncep.noaa.gov/static_files/public/bufr.tar
set(BUFR_URL "https://ftp.emc.ncep.noaa.gov/static_files/public")
if(${PROJECT_VERSION} VERSION_GREATER_EQUAL 11.6.0)
  set(BUFR_TAR "bufr-11.6.0.tgz")
else()
  set(BUFR_TAR "bufr.tar")
endif()

although perhaps it doesn't matter since neither bufr-11.7.0.tgz nor bufr.tar exist at that URL. I don't fully understand why only bufr-11.6.0.tgz versions exist at that URL but a generic bufr.tar is being referenced there.

The problem in compiling BUFRLIB version 11.7.0 on Derecho lies in the ctest step:

The following tests FAILED:
         13 - test_OUT_6_8 (Failed)
         16 - test_debufr_1 (Failed)
         17 - test_debufr_2 (Failed)
         19 - test_sinv (Failed)
Errors while running CTest
Output from these tests are in: /glade/p/ral/jntp/MET/MET_releases/12.0.0-beta1_oneapi/external_libs/bufrlib/NCEPLIBS-bufr-bufr_v11.7.0/build/Testing/Temporary/LastTest.l\
og

Please see:
LastTest.log
if you're interested.

Realizing I hadn't tried installing version 11.6.0 of BUFRLIB, I went ahead and tried that, and fortunately got a successful compilation of BUFRLIB using the new oneAPI compilers on Derecho.

@jbathegit, I see you are the code manager for this repo. Is this Discussions forum the appropriate place to post for help with BUFRLIB or is there another preferred way to request help?

[jbathegit]

Hello @jprestop. Thanks for reaching out to us, and I'm sorry to hear that you're encountering difficulties!

First off, and to answer your last question, we typically use the "Discussions" forum to bandy about ideas internally among all of the library developers, and we ask users to reach out to us with any questions or concerns by opening a new issue under the "Issues" tab. That will typically get it seen by a wider audience and therefore hopefully a more timely response, but no worries and I'll give you some initial feedback here.

At this point, we didn't expect that anyone would still be trying to build v11.5.0, because it's more than two and a half years old and we've fixed a number of bugs and added a number of useful new features to the library in a number of new releases since then. So we're really trying to wean people off of those older versions, and we have reached out to the WCOSS2 folks several times to ask them to link their "default" label to a newer version, but it's been slow trying to get any movement on that.

Along those lines, the bufr.tar was only intended for versions 11.5.0 and older, and any newer library version beyond 11.6.0 only needs the bufr-11.6.0.tgz. In other words, if you're trying to build 11.7.0 or later, then it should indeed be trying to pull the bufr-11.6.0.tgz file (i.e. that's not a mistake, and there really is no corresponding .tgz file for 11.7.0 ;-)

That said, and looking at your LastTest.log file, it looks like there was a mistake on our end in omitting the testoutput file "sinv.out" from the latest bufr-11.6.0.tgz. This happened because we recently changed the sinv utility and its associated tests in a more recent version of the library, and we apparently forgot to include the old "sinv.out" in the latest tarfile for backwards compatibility with testing earlier versions. I apologize for that, and I've now gone ahead and fixed that on the server, so it should work now for your 11.7.0 build if you go ahead and pull that tarfile again and rerun that test for that build.

However, that still leaves 3 other tests of your 11.7.0 build that apparently segfaulted, and I wasn't able to reproduce those locally so I'm a bit stuck. Admittedly we only have Intel 19.1.3.304 on our WCOSS2 system, whereas you apparently have a much newer version of Intel that you're trying to build with, so I'd probably need to be able to access the artifacts of those failed tests (including the .x files) on that particular machine in order to hook them up to a debugger or stack tracer on our end to try and narrow down where the segfaults are actually happening. And I'd probably also need you to try rebuilding your library with -DCMAKE_BUILD_TYPE=Debug as an added flag during your cmake step, in order to have any hope of seeing anything useful in a rerun test on our end.

I'm sorry I don't have a better answer at this point. We do test the library extensively on a number of platforms and OS variants including Intel, MacOS, Linux, etc., but again we don't have access to any machines running more recent versions of Intel, so that may well at least partially explain the problem. If you really need it, I can try to reconstruct a bufr.tar from some old records and put it back on the server, but again that wouldn't allow you to build and test anything beyond v11.5.0 of the library, and we are really trying to focus our limited support resources on newer versions.

[jprestop]

Thanks for your reply @jbathegit and for letting me know it is preferred to open a new issue and thanks for the feedback here.

Thank you also for reaching out to the WCOSS2 folks several times to ask them to link their "default" label to a newer version. I'm sorry to hear its been slow going, but I definitely understand. I would prefer to use 11.7.0 since it is available on WCOSS2 with the thought that eventually they'll move to that version as the default. Ideally, I'd like to grab an even later version, but due to the lack of availability on WCOSS2, I have to restrict our version to what is available there.

Thanks for fixing one of the cases that failed in 11.7.0. You mentioned that only Intel 19.1.3.304 is available on WCOSS2, but actually the oneapi compilers are available as " intel-oneapi/2022.2.0.262" on acorn, dogwood, and cactus (which I didn't know until recently after reaching out to Steven Earle to ask about it):

julie.prestopnik@dlogin02:~> module avail intel

----------------------------------------------------------------- WCOSS2 Core Applications and Compilers ------------------------------------------------------------------
   PrgEnv-intel/8.0.0    PrgEnv-intel/8.2.0        intel-classic/2022.2.0.262    intel/19.1.3.304
   PrgEnv-intel/8.1.0    PrgEnv-intel/8.3.3 (D)    intel-oneapi/2022.2.0.262

Of course, access to these compilers on WCOSS2 doesn't necessarily mean that you'll be able to reproduce the problem I experienced. I pulled the 11.7.0 tarfile again and reran using -DCMAKE_BUILD_TYPE=Debug as an added flag during the cmake step. You mentioned:

I'd probably need to be able to access the artifacts of those failed tests (including the .x files) on that particular machine in order to hook them up to a debugger or stack tracer on our end to try and narrow down where the segfaults are actually happening.

Can you please tell me where I can find those artifacts that you need? I'm not sure where to look. For now, I have pulled these files:
bufr.make.log
bufr.ctest.log
LastTest.log
I'm happy to grab others once I know where they are and what you need.

[jbathegit]

Sorry I got sidetracked on a different task, and thanks for the heads-up about the intel-oneapi/2022.2.0.262 compiler! I wasn't aware that was even an option on WCOSS2, so I just tried it myself with a clone of 11.7.0, but it almost immediately conked out with a weird error in the make step:

Scanning dependencies of target bufr_4_f
[  0%] Building Fortran object src/CMakeFiles/bufr_4_f.dir/modv_NFILES.f90.o
[  0%] Building Fortran object src/CMakeFiles/bufr_4_f.dir/moda_msgcwd.F.o
[  0%] Building Fortran object src/CMakeFiles/bufr_4_f.dir/modv_MAXJL.f.o
[  0%] Building Fortran object src/CMakeFiles/bufr_4_f.dir/moda_tables.F.o
[  0%] Building Fortran object src/CMakeFiles/bufr_4_f.dir/modv_MAXSS.f.o
[  0%] Building Fortran object src/CMakeFiles/bufr_4_f.dir/moda_usrint.F.o
[  0%] Building Fortran object src/CMakeFiles/bufr_4_f.dir/bufr_interface.f90.o
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x146f4ea]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x10f84dd]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x1156992]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x1156ac0]
/lib64/libc.so.6(+0x4ad00)[0x14ff989ded00]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0xe8d718]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x1189506]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x11842fc]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x11a89b3]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x11a9042]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x11ab3bf]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x11a89b3]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x11a9042]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x11a6722]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x11a89b3]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x11a5dba]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x11a89b3]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x1055476]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x1054c04]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0x11fb612]
/lib64/libc.so.6(__libc_start_main+0xef)[0x14ff989c924d]
/pe/intel/2022-2/compiler/2022.1.0/linux/bin-llvm/xfortcom[0xe701e9]

/lfs/h2/emc/obsproc/noscrub/jeff.ator/NCEPLIBS-bufr-TEMP11.7/NCEPLIBS-bufr-bufr_v11.7.0/src/bufr_interface.f90: error #5633: **Internal compiler error: segmentation violation signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report.  Note: File and line given may not be explicit cause of this error.
compilation aborted for /lfs/h2/emc/obsproc/noscrub/jeff.ator/NCEPLIBS-bufr-TEMP11.7/NCEPLIBS-bufr-bufr_v11.7.0/src/bufr_interface.f90 (code 3)
make[2]: *** [src/CMakeFiles/bufr_4_f.dir/build.make:4066: src/CMakeFiles/bufr_4_f.dir/bufr_interface.f90.o] Error 3
make[1]: *** [CMakeFiles/Makefile2:214: src/CMakeFiles/bufr_4_f.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

The only thing I changed here was swapping out the newer Intel compiler module for the older 19.1.3.304 version, so not sure if maybe there's something else I also need to change. At least based on a quick Google search, it looks like it may have been a compiler bug that wasn't fixed until a later version 2022.12.30.005, but that was also using ifx and I'm still only using ifort. So not sure if that's really the same thing, and I'll have to dig some more to try to figure that out.

In the meantime, one of the artifacts that might be useful is if you could maybe somehow share copies of your debufr and test_OUT_6_8.x executables - the ones that you were able to build with the oneAPI compiler but which were failing in ctest. Those executables should be in your build/utils and build/test directories, respectively, and maybe you could share them as email attachments, or post them somewhere on public ftp or some other server that I could access? My thinking was that if you were able to build those with the -DCMAKE_BUILD_TYPE=Debug settings on your end, then maybe(?) I could try to step through them with a debug tool on my end and figure out where and why they're segfaulting. Admittedly it may not work b/c it's a different machine and different compiler version, but it might be worth a shot b/c I'm not sure how else to try to troubleshoot this at this point since I wasn't able to reproduce those same errors locally.

[jbathegit]

Follow-up - I reached out to one of our WCOSS2 GDIT support folks, and he confirmed that the above error is something internal to the compiler, so not much we could do there other than open a ticket with Intel. But instead, he pointed me to a test installation of a newer version 2023.2.0 of the oneAPI compiler that he'd already built on acorn (i.e. the WCOSS2 test and development machine), and he asked me to try that. I did (with bufr v11.7.0), and everything built and ran fine with that version of the oneAPI compiler, and with no failed tests.

Could you please clarify what version of the oneAPI compiler you've been using in your tests? I'd definitely like to help you nail this down if we can - thanks!

[jprestop]

Hi @jbathegit. Apologies for the delayed response. I meant to follow up yesterday, but got caught up in other things. The machine I was testing on is down this week for maintenance so I can't get on until it is back up again. In the meantime, with regard to your question about what version of oneAPI I am using, according the module load command, I am using version 2023.0.0. It's great news that everything built well for bufr v11.7.0 with version 2023.2.0! I'll follow up more once I can get back on the machine. Side note, one of our system administrators just put version 2023.2.0.49397 on a machine that I have access to, so I may be able to try it out there sometime soon as well, but I have some other tasks I need to work on first. I'll follow up with more information for either machine once I am able to. Thank you so much for all of your help so far. I really appreciate it!

[jbathegit]

I know this has become a stale thread, but for the record any oneAPI compilation issues should now be fixed in the latest v12.1.0 release of the library. See #536 for more details, and so long as you're using OneAPI 2024.2 or later, you should be good to go now!

[jprestop]

@jbathegit Apologies for the delay here. I never got back to testing this and it fell off my radar. Thanks for letting me know about oneAPI compilation issues being resolved in the latest v12.1.0 release. That is great news. At some point, I'm sure the sys admins will completely remove the older compilers, so it's great to know that v12.1.0 should work with the oneAPI compilers. I appreciate you following up to let me know.