3DVAR and hybrid marineanlvar ctest fails when using soca @ 3dd464a

[RussTreadon-NOAA]

GDASApp successfully runs all C48mx500_3DVarAOWCDA and C48mx500_hybAOWCDA ctests when using GDASApp develop at 70ea55b. This GDASApp hash uses soca @ 4d51dca.

When stable-nightly testing run, it updates the following JEDI hashes

        modified:   sorc/fv3-jedi (new commits)
        modified:   sorc/ioda (new commits)
        modified:   sorc/oops (new commits)
        modified:   sorc/saber (new commits)
        modified:   sorc/soca (new commits)
        modified:   sorc/ufo (new commits)

The soca hash is updated to hash 3dd464a. See soca PR #1122 for details

The following GDASApp ctests fail when using the above updated JEDI hashes

        2043 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 (Failed)
        2044 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
        2045 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
        2055 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000 (Failed)
        2056 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
        2057 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000 (Failed)

The 3DVAR and hyb marineanlvar failures are the same

0: QC insitu_profile_argo salinity: 37507 passed out of 45599 observations.
 0:
 0: FATAL from PE     0: fms_io(register_restart_field_r2d): need to call fms_io_init
 0:
 0: Image              PC                Routine            Line        Source
 0: libifcoremt.so.5   000014CF35A9CDCB  tracebackqq_          Unknown  Unknown
 0: libsoca.so         000014CF271E1AC9  mpp_mod_mp_mpp_er          72  mpp_util_mpi.inc
 0: libsoca.so         000014CF2709EEE2  fms_io_mod_mp_reg        1689  fms_io.F90
 0: libsoca.so         000014CF26F2D921  Unknown               Unknown  Unknown

soca PR #1122 is a FMS IO bug fix. The addition of this bug fix breaks GDASApp marineanlvar.

Do we need to update marine DA yamls, parm files, or model input files in order to run GDASApp CI with soca at 3dd464a?

[RussTreadon-NOAA]

Attention @guillaumevernieres and @shlyaeva

[RussTreadon-NOAA]

Test

Retain updated jedi hashes except soca.

• fv3-jedi @ 9440a15
• ioda @ 9644fa1
• oops @ 60dd7b1
• saber @ b22a04
• ufo @ 9107231

Revert soca back to soca @ 4d51dca.

Rebuild GDASApp and rerun ctests. All tests passed

Test project /scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/stable/20250118/global-workflow/sorc/gdas.cd/build
        Start 1987: test_gdasapp_C96C48_ufs_hybatmDA
        Start 1968: test_gdasapp_C96C48_hybatmDA
        Start 2012: test_gdasapp_C96C48_hybatmaerosnowDA
        Start 2037: test_gdasapp_C48mx500_3DVarAOWCDA
        Start 2046: test_gdasapp_C48mx500_hybAOWCDA
        Start 2060: test_gdasapp_setup_atm_jjob_cycled_exp
        Start 1586: test_gdasapp_util_prepdata

...

133/134 Test #2003: test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402240000 ..................   Passed  477.28 sec
134/134 Test #2011: test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402240000 ..............   Passed  364.01 sec

100% tests passed, 0 tests failed out of 134

Label Time Summary:
gdas-utils    =  19.25 sec*proc (14 tests)
gdasapp       = 21081.66 sec*proc (102 tests)
script        = 21100.91 sec*proc (116 tests)

Total Test time (real) = 3606.08 sec

This success coupled with the failure of the stable-nightly update indicates that GDASApp marine DA fails when using soca at 3dd464a. This is the current head of soca develop.