Enable DA cycling on gaea #3032
Conversation
jswhit2
requested review from
WalterKolczynski-NOAA,
aerorahul,
CoryMartin-NOAA and
DavidHuber-NOAA
as code owners
|
c6 is basically the same - there are only subtle differences in some of the system libraries. This means that some of the executables compiled on c5 won't run on c6 (the model does, but GSI doesn't). So, the biggest obstacle to running on c6 will be to update all the build systems to have separate c5/c6 modulefiles. |
|
Thanks @jswhit2 for the explanation |
WalterKolczynski-NOAA
changed the title
There was a problem hiding this comment.
It appears this PR is specifically to enable cycling for the JEDI system? The atm-only GSI-based test at least appears to already be enabled on Gaea.
If this PR enables cycling with the JEDI system, those tests (in ci/cases/pr) should probably be tested and enabled.
env/GAEA.env
Outdated
| ;; | ||
| *) | ||
| # Some other job not yet defined here | ||
| echo "WARNING: The job step ${step} does not specify Hercules-specific resources" |
There was a problem hiding this comment.
| echo "WARNING: The job step ${step} does not specify Hercules-specific resources" | |
| echo "WARNING: The job step ${step} does not specify Gaea-specific resources" |
There was a problem hiding this comment.
@WalterKolczynski-NOAA I have been running the coupled atm/ocean 3dvar, so I wasn't aware it already worked for atm only DA. When running coupled DA without these mods, load_ufsda_modules.sh quits with the message "WARNING: UFSDA NOT SUPPORTED ON THIS PLATFORM".
|
Should coordinate with @AnilKumar-NOAA as well, who has been working on the Gaea port. |
|
Yes, I'll port and test on gaea but only after all recent OS upgrades on gaea PR's are merged. We are waiting on that. |
|
Hi @jswhit2 @WalterKolczynski-NOAA @CoryMartin-NOAA @AnilKumar-NOAA...Walter is correct, the C96_atm3DVar test case PR2885 was merged for Gaea-C5. However, with the recent C5 OS upgrade, all submodules within the global-workflow fail to build. After talking with Rahul, we decided to submit PRs to build submodules on both C5 and C6 and to follow the filename updates of the ufs-wx-model PR2448, e.g., having a gaeac5 and gaeac6 module files, having detect machine scripts return gaeac5 or gaeac6, etc. I have submitted PRs for Gaea-C5 and C6 builds for ufs-utils, gfs-utils, upp, gsi, gsi-monitor, and gsi-utils. I could set up a GDAS C5 and C6 build PR if you want/need. I'm in training full time through Thursday just to note. |
yes please, that would be great. I'd be happy to do the testing needed to make sure things work on c5/c6 once all the builds work. |
@jswhit2 Sounds good. I submitted an issue to build GDASApp on Gaea-C5 and C6 here. I will keep you updated on build progress. Is test data already set up on both systems? |
|
@jswhit As @CoryMartin-NOAA said, C6 build was already set up. I made adjustments to C5 module files and machine ID to work with how the global-workflow will be configured, i.e., using gaeac5 and gaeac6 throughout global-workflow and submodules. I have a branch here that builds on both C5 and C6 with those C5 updates. @jswhit Let me know if you want to run tests on this set up, or since the changes are minimal, I may just submit this PR for building. Thanks |
|
@jswhit Since your PR is for the global-workflow side, I submitted the GDAS PR1361 . |
…al-workflow (#1361) After the recent Gaea-C5 OS upgrade, GDASApp fails to build. This issue corrects Gaea-C5 build and updates the build scripts to conform to ufs-wx-model (following ufs-wx-model ufs-community/ufs-weather-model#2448) and eventual global-workflow updates. Refs NOAA-EMC/global-workflow 3011 NOAA-EMC/global-workflow#3011 Refs NOAA-EMC/global-workflow 3032 NOAA-EMC/global-workflow#3032 Resolves #1360
|
Is there anything you're waiting on me for, or are @jswhit and @DavidBurrows-NCO / @AnilKumar-NOAA still coordinating things? |
|
@DavidBurrows-NCO everything builds on c6, thanks. I updated this PR to include changes in global-workflow to build and run on both c5/c6. |
|
the data directories in /gpfs/f5/ufs-ard need to be mirrored on f6 though - @AnilKumar-NOAA do you have access to f6? |
|
@AnilKumar-NOAA @jswhit Glad to hear building on C6 worked. Cory did most of the heavy lifting. I am currently working on transferring /scratch1/NCEPDEV/global/glopara/data/ICSDIR and /scratch1/NCEPDEV/global/glopara/fix/ from Hera to C6. I can then quickly test some low resolution forecast-only type runs. For the DA runs, we'll need some dump data but currently Hera has 111TB, which I don't want to store on bil-fire8 in case EPIC loses access to that group at some point. Currently, I plan to bring in the dump data that supports EMC's ci/cases/pr/C96_atm3DVar.yaml test case with idate=2021122018 and edate=2021122106. @jswhit Any particular test dates you are interested in? |
|
@DavidBurrows-NCO I am currently testing 2021032400-2021032600, so if you could bring those dumps over too that would be great. I'm running the coupled 3dvar, and my test yaml is at /ncrc/home2/Jeffrey.S.Whitaker/C192mx025_3DVarAOWCDA.yaml. |
@jswhit Is this the subset you are expecting? I looked into the subdirectories and notice that /scratch1/NCEPDEV/global/glopara/dump/gfs.20210324/06 is missing. @aerorahul Should I open an issue for this? |
probably. Tagging @KateFriedman-NOAA as she keeps the dump dataset synced |
I'll get that filled in on Hera...stay tuned... |
|
The 2021032406 dump files for gdas and gfs are now available on Hera. Does anyone need this cycle on WCOSS2 or MSU as well? |
Yes, that should do it. I also need the ocean obs for the same time period (on hera they are at /scratch1/NCEPDEV/da/common/, I've already copied them to c5 and c6 /gpfs/f5/nggps_psd/proj-shared/Jeffrey.S.Whitaker/da/common/, /gpfs/f6/ira-da/proj-shared/Jeffrey.S.Whitaker/da/common/) |
Description
currently DA cycling on gaea is disabled. These changes remove that restriction.
Type of change
New feature (adds functionality)
Change characteristics
How has this been tested?
ran a cycled test with coupled 3dvar (0.5 degree atmosphere and 0.25 degree ocean).
Checklist