Feature/levels metadata #28

phlndrwd · 2023-11-17T16:34:49Z

Addition of vertical meta/data to NetCDF files for outputs that adopt JEDI variable conventions.

This PR will be part of a coordinated merge with:

LFRic-Lite https://github.com/JCSDA-internal/lfric-lite-jedi/pull/376

Test outputs (LFRic-Lite, LFRic-JEDI, & MONIO): http://fcm1/cylc-review/taskjobs/punderwo/?suite=levels_metadata_01

…adata.

…rings.

… applicable, and dropped all but essential metadata between writing of globalised field data. Updated comments to suit.

…mension. Dropped zeroth level processing for JEDI-only output files. Const correctness in some Metadata functions.

…ant development code that was also causing an issue.

…fields appropriately. Changed position if LFRic and JEDI in eVariableConventions enum so that LFRic is at position 1 and can be used instead of bool, if necessary.

mo-joshuacolclough · 2023-11-20T13:28:30Z

Failing on develop with LFRic-JEDI:

Simplest test that is failing is the ctest_lfricjedi_xios_monio_comparison test. Currently looking into it...

Full output: http://fcm1/cylc-review/view/jcolclou?&suite=check_monio_change__201123&no_fuzzy_time=0&path=log/job/1/ctest_lfricjedi_xios_monio_comparison_lfric__spice_gnu/02/job.out

Summary of failure:

10: 20231120131301.016+0000testCompareXIOSMONIOStateRead> Reading XIOS(0) done.
10: testCompareXIOSMONIOStateRead> xiosState: 
10: 
10: ---------- State print ----------
10: Valid time: 2018-04-14T21:00:00Z
10: Number of variables: 4
10: Locally owned data num: 30384
10: 
10: 'eastward_wind'
10:   Min=-5.16192068802978099e+01, Max=8.38294119533246800e+01, RMS=1.64436981523366548e+01
10: 'northward_wind'
10:   Min=-5.18142921369839655e+01, Max=4.71832627877766768e+01, RMS=7.17428459853106837e+00
10: 'potential_temperature'
10:   Min=2.37215478585606320e+02, Max=5.73111180066490306e+03, RMS=1.15127835739124657e+03
10: 'skin_temperature'
10:   Min=2.15627294832477958e+02, Max=3.12638033249410739e+02, RMS=2.87465032112199594e+02
10: ---------------------------------
10: 
10: testCompareXIOSMONIOStateRead> Reading MONIO(0)...
10: testCompareXIOSMONIOStateRead> Reading MONIO(0) done.
10: testCompareXIOSMONIOStateRead> monioState: 
10: 
10: ---------- State print ----------
10: Valid time: 2018-04-14T21:00:00Z
10: Number of variables: 4
10: Locally owned data num: 30384
10: 
10: 'eastward_wind'
10:   Min=-5.16192068802978099e+01, Max=8.38294119533246800e+01, RMS=1.64436981523366548e+01
10: 'northward_wind'
10:   Min=-5.18142921369839655e+01, Max=4.71832627877766768e+01, RMS=7.17428459853106837e+00
10: 'potential_temperature'
10:   Min=2.37215478585606320e+02, Max=4.80996841150037835e+03, RMS=9.46595593171428391e+02
10: 'skin_temperature'
10:   Min=2.15627294832477958e+02, Max=3.12638033249410739e+02, RMS=2.87465032112199594e+02
10: ---------------------------------
10: 
10: Test "lfricjedi/testCompareXIOSMONIOStateRead" failed with unhandled eckit::Exception: Assertion failed: testCompareXIOSMONIOStateRead> XIOS and MONIO reads do not match. in testCompareXIOSMONIOStateRead, line 124 of /home/h02/jcolclou/cylc-run/check_monio_change__201123/share/mo-bundle/lfric-jedi/test/bespoke/IO/XIOSMONIOComparison.h

It seems that the potential_temperature values being read in no longer match what XIOS is reading. That is the theta field which is a full_level field missing the surface.

mo-joshuacolclough · 2023-11-20T13:31:04Z

Failing on develop with LFRic-JEDI:

Simplest test that is failing is the ctest_lfricjedi_xios_monio_comparison test. Currently looking into it...

Full output: http://fcm1/cylc-review/view/jcolclou?&suite=check_monio_change__201123&no_fuzzy_time=0&path=log/job/1/ctest_lfricjedi_xios_monio_comparison_lfric__spice_gnu/02/job.out

Summary of failure:

10: 20231120131301.016+0000testCompareXIOSMONIOStateRead> Reading XIOS(0) done.
10: testCompareXIOSMONIOStateRead> xiosState: 
10: 
10: ---------- State print ----------
10: Valid time: 2018-04-14T21:00:00Z
10: Number of variables: 4
10: Locally owned data num: 30384
10: 
10: 'eastward_wind'
10:   Min=-5.16192068802978099e+01, Max=8.38294119533246800e+01, RMS=1.64436981523366548e+01
10: 'northward_wind'
10:   Min=-5.18142921369839655e+01, Max=4.71832627877766768e+01, RMS=7.17428459853106837e+00
10: 'potential_temperature'
10:   Min=2.37215478585606320e+02, Max=5.73111180066490306e+03, RMS=1.15127835739124657e+03
10: 'skin_temperature'
10:   Min=2.15627294832477958e+02, Max=3.12638033249410739e+02, RMS=2.87465032112199594e+02
10: ---------------------------------
10: 
10: testCompareXIOSMONIOStateRead> Reading MONIO(0)...
10: testCompareXIOSMONIOStateRead> Reading MONIO(0) done.
10: testCompareXIOSMONIOStateRead> monioState: 
10: 
10: ---------- State print ----------
10: Valid time: 2018-04-14T21:00:00Z
10: Number of variables: 4
10: Locally owned data num: 30384
10: 
10: 'eastward_wind'
10:   Min=-5.16192068802978099e+01, Max=8.38294119533246800e+01, RMS=1.64436981523366548e+01
10: 'northward_wind'
10:   Min=-5.18142921369839655e+01, Max=4.71832627877766768e+01, RMS=7.17428459853106837e+00
10: 'potential_temperature'
10:   Min=2.37215478585606320e+02, Max=4.80996841150037835e+03, RMS=9.46595593171428391e+02
10: 'skin_temperature'
10:   Min=2.15627294832477958e+02, Max=3.12638033249410739e+02, RMS=2.87465032112199594e+02
10: ---------------------------------
10: 
10: Test "lfricjedi/testCompareXIOSMONIOStateRead" failed with unhandled eckit::Exception: Assertion failed: testCompareXIOSMONIOStateRead> XIOS and MONIO reads do not match. in testCompareXIOSMONIOStateRead, line 124 of /home/h02/jcolclou/cylc-run/check_monio_change__201123/share/mo-bundle/lfric-jedi/test/bespoke/IO/XIOSMONIOComparison.h

It seems that the potential_temperature values being read in no longer match what XIOS is reading. That is the theta field which is a full_level field missing the surface.

Requires a small change in LFRic-JEDI to handle the new metadata

…ced by MONIO were not interpretted as LFRic files.

…on' in Monio class.

mo-joshuacolclough

Thanks looks good. Haven't re-tested, but we've discussed previous issues and my understanding is that those are now fixed. Cheers

mo-joshuacolclough · 2023-11-20T13:44:55Z

src/monio/AtlasWriter.cc

-                              consts::kNamingConventions[consts::eJediNaming];
+  std::string variableConvention =
+      isLfricConvention == true ? consts::kNamingConventions[consts::eLfricConvention] :
+                              consts::kNamingConventions[consts::eJediConvention];


Indent slightly off compared to above line

mo-joshuacolclough · 2023-11-20T13:45:47Z

src/monio/Metadata.cc

@@ -373,7 +373,7 @@ void monio::Metadata::deleteVariable(const std::string& varName) {

 void monio::Metadata::clear() {
  oops::Log::debug() << "Metadata::clear()" << std::endl;
-  dimensions_.clear();
+  // dimensions_.clear();  // Dimensions are required for correct writing of subsequent variables.


Would be good to remove commented out code if it's not needed anymore

phlndrwd added 14 commits November 14, 2023 17:54

Added two additional vertical configurations items to struct FieldMet…

92a3f7d

…adata.

Updated YAMLs for additional LFRic and JEDI vertical configuration st…

0683ab4

…rings.

Removed duplicate line in FieldSetWrite.

155844c

Removed redundant file closures from write functions in the Monio class.

8e7ee1f

Switched to use of 'variable convention' where applicable.

e1903e8

Further changes for switch to use of 'variable convention'.

77363a8

Progress towards addition of JEDI vertical meta/data.

fce6414

Reinstated known working code.

bf5db11

Updated comments and alignment in places.

29a8a91

Included addition of metadata variables for JEDI increment files.

a3ebfe8

Included addition of metadata variable attributes and dimensions.

c746d94

Enhanced code efficiency by making single call to add JEDI data where…

1affafb

… applicable, and dropped all but essential metadata between writing of globalised field data. Updated comments to suit.

Using vertical configuration from FieldMetadata to derive vertical di…

dcc5524

…mension. Dropped zeroth level processing for JEDI-only output files. Const correctness in some Metadata functions.

Fixed bug whereby non-existant dimension was accessed. Removed redund…

ee9eca6

…ant development code that was also causing an issue.

phlndrwd added the do not merge label Nov 17, 2023

phlndrwd self-assigned this Nov 17, 2023

phlndrwd requested a review from mo-joshuacolclough November 17, 2023 16:38

Passed isLFRicConvention flag to AtlasReader to treat LFRic and JEDI …

3f68160

…fields appropriately. Changed position if LFRic and JEDI in eVariableConventions enum so that LFRic is at position 1 and can be used instead of bool, if necessary.

phlndrwd added 2 commits November 22, 2023 13:43

Removed subtle bug whereby LFRic-compatible files that were not produ…

b09c478

…ced by MONIO were not interpretted as LFRic files.

Changed remainining usages of 'namingConvention' to 'variableConventi…

ab0fd0b

…on' in Monio class.

phlndrwd removed the do not merge label Nov 22, 2023

Removal of using std::iota from Constants.

a496d79

phlndrwd marked this pull request as ready for review November 22, 2023 18:12

mo-joshuacolclough approved these changes Nov 27, 2023

View reviewed changes

phlndrwd merged commit 083e3c1 into develop Nov 29, 2023
4 checks passed

phlndrwd deleted the feature/levels_metadata branch December 1, 2023 13:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/levels metadata #28

Feature/levels metadata #28

phlndrwd commented Nov 17, 2023 •

edited

Loading

mo-joshuacolclough commented Nov 20, 2023

mo-joshuacolclough commented Nov 20, 2023

mo-joshuacolclough left a comment

mo-joshuacolclough Nov 20, 2023

mo-joshuacolclough Nov 20, 2023

Feature/levels metadata #28

Feature/levels metadata #28

Conversation

phlndrwd commented Nov 17, 2023 • edited Loading

mo-joshuacolclough commented Nov 20, 2023

mo-joshuacolclough commented Nov 20, 2023

mo-joshuacolclough left a comment

Choose a reason for hiding this comment

mo-joshuacolclough Nov 20, 2023

Choose a reason for hiding this comment

mo-joshuacolclough Nov 20, 2023

Choose a reason for hiding this comment

phlndrwd commented Nov 17, 2023 •

edited

Loading