Auto merge partitioned netcdf files

[alperaltuntas]

This is one of three PRs to enable automated parallel IO in MOM6 within CESM by making use of the existing parallel IO implementation that comes with FMS. These series of PRs (1) enable FMS parallel IO, (2) make sure the IO PE Layout is compatible with auto-land block elimination (i.e., masking), and (3) that partitioned files are automatically merged after a run is completed.

The major change in this PR is the introduction of mppnccombine tool into the FMS diag_manager. When an IO_LAYOUT is specified, FMS now automatically merges the partitioned files when diag manager closes the open netcdf files. For domain files, this is done in a round robin fashion, where each IO PE takes care of the auto-merging of a history file stream. For section files, the IO PE with the smallest PE index takes care of the merging. The auto merge feature is enabled by the newly added auto_merge_nc namelist parameter.

testing: aux_mom.derecho
status: b4b, except for masking in static files due to changing compute layout.
Performance: varies depending on the resolution, pe count and IO layout, from no enhancement or degradation to more than 50% enhancement.

This PR should be evaluated in conjunction with:
NCAR/MOM6#342
ESCOMP/MOM_interface#235

[alperaltuntas]

Mentioning @mnlevy1981 and @gustavo-marques.

[jedwards4b]

What is the suggested MOM IO strategy for fully coupled compsets? Is this something that can be set as default or does it require user modifications?

[jedwards4b]

I think that it would have been better to integrate MOM IO with PIO as with other component models, but I understand the motivation for doing it this way.

[alperaltuntas]

@jedwards4b I think this approach is lightweight and more maintainable in the long run. I do have an FMS branch where I integrated PIO into the FMS diagnostic manager. That integration is mostly complete, with two remaining issues: compatibility with auto-masking and handling of section files.

However, that branch introduces significant changes, over 15,000 new lines of code, which makes me concerned about maintainability and potential fragility. Given the scope of these modifications, I opted for this simpler approach. That said, I’m happy to continue pursuing the full PIO integration if I can get support for resolving the remaining issues and, more importantly, assistance with ongoing maintenance, including porting to newer versions of FMS as needed.

Here’s the branch with the PIO integration:
dev/ncar...alperaltuntas:FMS:dev/ncar_add_pio