MOM5 can't output _only_ gyre/overturning diagnostics in a file

[dougiesquire]

There are a set of MOM5 diagnostics included in both the standard and detailed diagnostic profiles related to the gyre/overturning. These have native diagnostic names of the form *_merid_flux_*.

For some (yet unknown) reason, these cannot be the only diagnostic(s) in an output file. Doing so results in the diagnostics silently being incorrect. For example, our ACCESS-ESM1.5 configurations currently output each gyre/overturning diagnostic in a separate file, giving outputs like:

rather than the correct:

Simply adding another non-gyre/overturning diagnostic to the output file fixes the issue.

This is a bug in MOM5/FMS and it should be fixed there. But in the meantime I propose as a workaround that we bundle all the gyre/overturning diagnostics into a single file and include one other diagnostic to this file to prevent the issue (e.g. geolat_c since it's at least related to the gyre/overturning diagnostics). While we're at it, we could also bundle together the scalar output (as is done for ACCESS-OM2) since it's a bit onerous to have these all in separate (tiny) files.

[aidanheerdegen]

Make sure we bundle all diagnostics changes into a single PR

#72

[dougiesquire]

Note, in some circumstances, MOM5 crashes with an FPE rather than silently giving the wrong answers - e.g. if I build ACCESS-ESM1.5 from the MOM5 master branch rather than the access-esm1.5 branch.

[gadi:1889444:0:1889444] Caught signal 8 (Floating point exception: floating-point overflow)
==== backtrace (tid:1889444) ====
 0 0x0000000000012d20 __funlockfile()  :0
 1 0x00000000000b1430 nc4_convert_type()  ???:0
 2 0x00000000000a5e57 NC4_put_vars()  ???:0
 3 0x00000000000a5696 NC4_put_vara()  ???:0
 4 0x00000000000290d0 nc_put_vara_double()  ???:0
 5 0x000000000001ecf0 nf_put_vara_double_()  /a/path/spack-src/fortran/nf_varaio.F90:427
 6 0x0000000001caa3c7 mpp_io_mod_mp_mpp_write_r4d_()  /a/path/spack-src/src/shared/mpp/include/mpp_write_2Ddecomp.h:143
 7 0x0000000001f015b7 diag_util_mod_mp_diag_data_out_()  /a/path/spack-src/src/shared/diag_manager/diag_util.F90:2436
 8 0x0000000001ee98c4 diag_manager_mod_mp_writing_field_()  /a/path/spack-src/src/shared/diag_manager/diag_manager.F90:3498
 9 0x0000000001ed745f diag_manager_mod_mp_send_data_3d_()  /a/path/spack-src/src/shared/diag_manager/diag_manager.F90:2008
10 0x0000000001ec57bd diag_manager_mod_mp_send_data_1d_()  /a/path/spack-src/src/shared/diag_manager/diag_manager.F90:1495
11 0x0000000000a8bd90 ocean_tracer_advect_mod_mp_gyre_overturn_diagnose_()  /a/path/spack-src/src/mom5/ocean_tracers/ocean_tracer_advect.F90:7389
12 0x0000000000acb0c6 ocean_tracer_advect_mod_mp_advect_tracer_sweby_all_()  /a/path/spack-src/src/mom5/ocean_tracers/ocean_tracer_advect.F90:4500
13 0x0000000000b0d60d ocean_tracer_advect_mod_mp_horz_advect_tracer_()  /a/path/spack-src/src/mom5/ocean_tracers/ocean_tracer_advect.F90:2079
14 0x00000000012038f2 ocean_tracer_mod_mp_update_ocean_tracer_()  /a/path/spack-src/src/mom5/ocean_tracers/ocean_tracer.F90:2300
15 0x000000000043c471 ocean_model_mod_mp_update_ocean_model_()  /a/path/spack-src/src/mom5/ocean_core/ocean_model.F90:1869
16 0x0000000000414c01 MAIN__()  /a/path/spack-src/src/access/accesscm_coupler/ocean_solo.F90:450
17 0x000000000040fc22 main()  ???:0
18 0x000000000003a7e5 __libc_start_main()  ???:0
19 0x000000000040fb2e _start()  ???:0

I haven't tracked down whether this is due to build differences (e.g. compiler flags) or source code differences.

[blimlim]

Thanks for spotting this @dougiesquire. Do these changes need someone to put them together? Happy to make a PR if they do.

[dougiesquire]

@blimlim I have these changes already in a test configuration for ESM1.6, so I can open a PR and then ping you for a review?

[blimlim]

That sounds great!