Cell indices for fields like cellsOnCell different in output for non-existent cells

[climbfuji]

Fields such as cellsOnCell have dimensions maxEdges and nCells, but only the values cellsOnCell(j,iCell) for j between 1 and nCellsOnCell(iCell) are used (iCell=1,nCells).

The issue here is that when writing these fields to disk, the unused values cellsOnCell(j>nCellsOnCell(iCell),iCell) differ depending on the parallelisation. In the mesh file, the unused values are set to nCells+1 (nCells is the global number of cells).

In the routine prewrite_reindex, these values are set to nCells+1, but nCells here is obtained from:

call mpas_pool_get_dimension(indexToCellID % block % dimensions, 'nCells', nCells)

This nCells is the task-local number of cells in indexToCellID and thus different from the global nCells. One solution could be to pass in domain % blocklist % dimensions and obtain the correct nCellsGlobal from there. However, this requires changing all interfaces up to mpas_stream_mgr_write, which means changes to every core.

I also found that mpas_io_streams makes use of a totalDimSize attribute of the field_list_type, however it seems wrong to me to try to use this, since it is one layer below mpas_stream_manager.

Another option could be to extend the mpas field types (field5DReal etc) to contain a totalDimSize (set to -999 or -1 if isDecomposed is false, and to the correct value otherwise). This could then be queried using mpas_pool_get_field_info (requires extending the field info type accordingly). And at some point during the definition of the fields during model startup, the value must be set correctly. This could happen in mpas_block_creator_finalize_block_phase2, I guess.

This "incorrect" behaviour doesn't change any of the results, since these values are not used; however, it causes problems when using ncdiff to compare netCDF files and then query maximum/minimum values or values different from zero.

This does not only affect cellsOnCell, but basically all fields that contain maxEdges as dimension.

[climbfuji]

Another option could be to set these unused fields to a missing/fill value (yet to be defined for MPAS variables), but then the mesh generation codes should do so as well, and existing meshes should be updated?

[akturner]

I've also noticed this behavior and fixing it would be a good thing to do.