Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix previous restart paths for ensemble #1772

Merged
merged 1 commit into from
Aug 4, 2023
Merged

Fix previous restart paths for ensemble #1772

merged 1 commit into from
Aug 4, 2023

Conversation

WalterKolczynski-NOAA
Copy link
Contributor

Description

GEFS was failing because the rCDUMP in the forecast job was hard-coded to enkfgdas. This resulted in always looking in that directory for previous cycle restart files. This was not caught previously because the manual staging method was copying the gdas directories as well, masking the problem.

Resolves #1771

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • Forecast-only GEFS test on Hera
  • Forecast-only GEFS test on Orion

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes

@WalterKolczynski-NOAA
Fix previous restart paths for ensemble
8750d84 
GEFS was failing because the rCDUMP in the forecast job was hard-
coded to enkfgdas. This resulted in always looking in that directory
for previous cycle restart files. This was not caught previously
because the manual staging method was copying the gdas directories
as well, masking the problem.

Resolves NOAA-EMC#1771
@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Aug 4, 2023
@WalterKolczynski-NOAA WalterKolczynski-NOAA self-assigned this Aug 4, 2023
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Aug 4, 2023
@emcbot
Copy link

emcbot commented Aug 4, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Aug  4 08:40:58 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug  4 08:45:35 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug  4 09:09:38 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:42 UTC 2023 for experiment C48_ATM_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:45 UTC 2023 for experiment C48_S2S_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:48 UTC 2023 for experiment C96C48_hybatmDA_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:51 UTC 2023 for experiment C96_atm3DVar_8750d841

@emcbot emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Aug 4, 2023
@emcbot
Copy link

emcbot commented Aug 4, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug  4 03:40:43 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug  4 03:42:42 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug  4 04:10:10 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 04:10:17 CDT 2023 for experiment C48_ATM_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 04:10:20 CDT 2023 for experiment C48_S2S_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 04:10:23 CDT 2023 for experiment C96_atm3DVar_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 04:10:27 CDT 2023 for experiment C96C48_hybatmDA_8750d841

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Aug 4, 2023
@emcbot
Copy link

emcbot commented Aug 4, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug  4 03:40:43 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug  4 03:42:42 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug  4 04:10:10 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 04:10:17 CDT 2023 for experiment C48_ATM_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 04:10:20 CDT 2023 for experiment C48_S2S_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 04:10:23 CDT 2023 for experiment C96_atm3DVar_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 04:10:27 CDT 2023 for experiment C96C48_hybatmDA_8750d841
Experiment C48_ATM_8750d841 Terminated: *FAILED*
Experiment C48_ATM_8750d841 Terminated with 7 tasks failed at Fri Aug  4 04:40:35 CDT 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1772/RUNTESTS/COMROT/C48_ATM_8750d841/logs/2021032312/gfsfcst.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1772/RUNTESTS/COMROT/C48_ATM_8750d841/logs/2021032312/gfspost_f027.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1772/RUNTESTS/COMROT/C48_ATM_8750d841/logs/2021032312/gfspost_f030.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1772/RUNTESTS/COMROT/C48_ATM_8750d841/logs/2021032312/gfspost_f033.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1772/RUNTESTS/COMROT/C48_ATM_8750d841/logs/2021032312/gfspost_f036.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1772/RUNTESTS/COMROT/C48_ATM_8750d841/logs/2021032312/gfspost_f039.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1772/RUNTESTS/COMROT/C48_ATM_8750d841/logs/2021032312/gfspost_f042.log

@emcbot
Copy link

emcbot commented Aug 4, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Aug  4 08:40:58 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug  4 08:45:35 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug  4 09:09:38 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:42 UTC 2023 for experiment C48_ATM_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:45 UTC 2023 for experiment C48_S2S_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:48 UTC 2023 for experiment C96C48_hybatmDA_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:51 UTC 2023 for experiment C96_atm3DVar_8750d841
Experiment C48_S2S_8750d841 completed: *SUCCESS*
Experiment C48_S2S_8750d841 Completed at Fri Aug  4 09:51:11 UTC 2023
with 18 successfully completed jobs

@emcbot
Copy link

emcbot commented Aug 4, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Aug  4 08:40:58 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug  4 08:45:35 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug  4 09:09:38 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:42 UTC 2023 for experiment C48_ATM_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:45 UTC 2023 for experiment C48_S2S_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:48 UTC 2023 for experiment C96C48_hybatmDA_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:51 UTC 2023 for experiment C96_atm3DVar_8750d841
Experiment C48_S2S_8750d841 completed: *SUCCESS*
Experiment C48_S2S_8750d841 Completed at Fri Aug  4 09:51:11 UTC 2023
with 18 successfully completed jobs
Experiment C48_ATM_8750d841 completed: *SUCCESS*
Experiment C48_ATM_8750d841 Completed at Fri Aug  4 10:21:16 UTC 2023
with 48 successfully completed jobs

@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed label Aug 4, 2023
@emcbot
Copy link

emcbot commented Aug 4, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Aug  4 08:40:58 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug  4 08:45:35 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug  4 09:09:38 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:42 UTC 2023 for experiment C48_ATM_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:45 UTC 2023 for experiment C48_S2S_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:48 UTC 2023 for experiment C96C48_hybatmDA_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:51 UTC 2023 for experiment C96_atm3DVar_8750d841
Experiment C48_S2S_8750d841 completed: *SUCCESS*
Experiment C48_S2S_8750d841 Completed at Fri Aug  4 09:51:11 UTC 2023
with 18 successfully completed jobs
Experiment C48_ATM_8750d841 completed: *SUCCESS*
Experiment C48_ATM_8750d841 Completed at Fri Aug  4 10:21:16 UTC 2023
with 48 successfully completed jobs
Experiment C96C48_hybatmDA_8750d841 completed: *SUCCESS*
Experiment C96C48_hybatmDA_8750d841 Completed at Fri Aug  4 12:27:10 UTC 2023
with 151 successfully completed jobs

@emcbot
Copy link

emcbot commented Aug 4, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Aug  4 08:40:58 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug  4 08:45:35 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug  4 09:09:38 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:42 UTC 2023 for experiment C48_ATM_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:45 UTC 2023 for experiment C48_S2S_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:48 UTC 2023 for experiment C96C48_hybatmDA_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:51 UTC 2023 for experiment C96_atm3DVar_8750d841
Experiment C48_S2S_8750d841 completed: *SUCCESS*
Experiment C48_S2S_8750d841 Completed at Fri Aug  4 09:51:11 UTC 2023
with 18 successfully completed jobs
Experiment C48_ATM_8750d841 completed: *SUCCESS*
Experiment C48_ATM_8750d841 Completed at Fri Aug  4 10:21:16 UTC 2023
with 48 successfully completed jobs
Experiment C96C48_hybatmDA_8750d841 completed: *SUCCESS*
Experiment C96C48_hybatmDA_8750d841 Completed at Fri Aug  4 12:27:10 UTC 2023
with 151 successfully completed jobs
Experiment C96_atm3DVar_8750d841 completed: *SUCCESS*
Experiment C96_atm3DVar_8750d841 Completed at Fri Aug  4 13:12:20 UTC 2023
with 89 successfully completed jobs

@emcbot emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Aug 4, 2023
@emcbot
Copy link

emcbot commented Aug 4, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Aug  4 08:40:58 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug  4 08:45:35 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug  4 09:09:38 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:42 UTC 2023 for experiment C48_ATM_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:45 UTC 2023 for experiment C48_S2S_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:48 UTC 2023 for experiment C96C48_hybatmDA_8750d841
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug  4 09:09:51 UTC 2023 for experiment C96_atm3DVar_8750d841
Experiment C48_S2S_8750d841 completed: *SUCCESS*
Experiment C48_S2S_8750d841 Completed at Fri Aug  4 09:51:11 UTC 2023
with 18 successfully completed jobs
Experiment C48_ATM_8750d841 completed: *SUCCESS*
Experiment C48_ATM_8750d841 Completed at Fri Aug  4 10:21:16 UTC 2023
with 48 successfully completed jobs
Experiment C96C48_hybatmDA_8750d841 completed: *SUCCESS*
Experiment C96C48_hybatmDA_8750d841 Completed at Fri Aug  4 12:27:10 UTC 2023
with 151 successfully completed jobs
Experiment C96_atm3DVar_8750d841 completed: *SUCCESS*
Experiment C96_atm3DVar_8750d841 Completed at Fri Aug  4 13:12:20 UTC 2023
with 89 successfully completed jobs

aerorahul
aerorahul approved these changes Aug 4, 2023
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests confirm that the current cycled system with RUN=enkfgdas works.
There is no test for the ensemble forecast when RUN=enkfgfs.
However, there is no need for running and ensemble forecast in the early cycle, as the early cycle is only to provide ICs for GEFS (RUN=gefs)

The changes look good to me.

If possible, a test should be added for the gefs experiment ASAP, even if it means hacking the manual staging in setup_expt.py temporarily.

@aerorahul aerorahul mentioned this pull request Aug 4, 2023
Merged
9 tasks
@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit 570206a into NOAA-EMC:develop Aug 4, 2023
4 checks passed
AnilKumar-NOAA
AnilKumar-NOAA reviewed Aug 8, 2023
View reviewed changes
Copy link
Contributor

@AnilKumar-NOAA AnilKumar-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve

@WalterKolczynski-NOAA WalterKolczynski-NOAA deleted the hotfix/gefs_rcdump branch August 8, 2023 18:12
AnilKumar-NOAA
AnilKumar-NOAA reviewed Aug 10, 2023
View reviewed changes
scripts/exgdas_enkf_fcst.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updates!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AnilKumar-NOAA AnilKumar-NOAA AnilKumar-NOAA left review comments

@aerorahul aerorahul aerorahul approved these changes

Assignees

@WalterKolczynski-NOAA WalterKolczynski-NOAA

Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Incorrect paths used for previous restart directories
4 participants
@WalterKolczynski-NOAA @emcbot @aerorahul @AnilKumar-NOAA