Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do Not Merge: Test ufs-utils from PR 815 #1818

Closed
wants to merge 1 commit into from
Closed

Do Not Merge: Test ufs-utils from PR 815 #1818

wants to merge 1 commit into from

Conversation

aerorahul
Copy link
Contributor

Description

This is a draft PR to test ufs-community/UFS_UTILS#815

Addresses global-workflow issue: #1775

DO NOT MERGE

@aerorahul aerorahul added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Aug 22, 2023
@github-actions
Copy link

Link to ReadTheDocs sample build for this PR can be found at:
https://global-workflow--1818.org.readthedocs.build/en/1818

@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Aug 22, 2023
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Aug 22, 2023
@aerorahul aerorahul mentioned this pull request Aug 22, 2023
Merged
6 tasks
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Aug 22, 2023
@TerrenceMcGuinness-NOAA
Copy link
Collaborator

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c

@emcbot emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Aug 22, 2023
@emcbot
Copy link

emcbot commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Tue Aug 22 13:44:22 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 13:46:43 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 14:16:31 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:37 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:40 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:43 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:45 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:49 CDT 2023 for experiment C96C48_hybatmDA_2974811c

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs

@emcbot
Copy link

emcbot commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Tue Aug 22 13:44:22 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 13:46:43 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 14:16:31 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:37 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:40 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:43 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:45 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:49 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 15:00:34 CDT 2023
with 18 successfully completed jobs

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 20:27:08 UTC 2023
with 48 successfully completed jobs

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 20:27:08 UTC 2023
with 48 successfully completed jobs
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 20:36:14 UTC 2023
with 18 successfully completed jobs

@emcbot
Copy link

emcbot commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Tue Aug 22 13:44:22 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 13:46:43 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 14:16:31 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:37 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:40 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:43 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:45 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:49 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 15:00:34 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 15:36:20 CDT 2023
with 48 successfully completed jobs

@emcbot
Copy link

emcbot commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Tue Aug 22 13:44:22 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 13:46:43 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 14:16:31 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:37 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:40 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:43 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:45 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:49 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 15:00:34 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 15:36:20 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 15:52:09 CDT 2023
with 3 successfully completed jobs

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Aug 22, 2023
@emcbot
Copy link

emcbot commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Tue Aug 22 13:44:22 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 13:46:43 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 14:16:31 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:37 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:40 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:43 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:45 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:49 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 15:00:34 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 15:36:20 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 15:52:09 CDT 2023
with 3 successfully completed jobs
Experiment C96_atm3DVar_2974811c Terminated: *FAILED*
Experiment C96_atm3DVar_2974811c Terminated with 2 tasks failed at Tue Aug 22 16:16:08 CDT 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1818/RUNTESTS/COMROT/C96_atm3DVar_2974811c/logs/2021122100/gdasanal.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1818/RUNTESTS/COMROT/C96_atm3DVar_2974811c/logs/2021122100/gfsanal.log

@aerorahul
Copy link
Contributor Author

@CatherineThomas-NOAA @GeorgeGayno-NOAA
Running with PR 815 from ufs-utils resulted in a failure in the GSI analysis jobs on Orion for both the GDAS and GFS cycles.
I don't see an obvious error message and am suspecting a glitch on Orion.
I will let the tests on Hera continue to see if they complete successfully.

@aerorahul
Copy link
Contributor Author

@CatherineThomas-NOAA @GeorgeGayno-NOAA
I checked the tests on Hera and they do seem to be progressing without errors with the UFS-utils from PR 815. I am sure now the failures on Orion in the GSI analysis jobs were a result of something with the system. Looking at the timestamps on those failed logs, they appear to be at the same time, likely confirming my suspicion.

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 20:27:08 UTC 2023
with 48 successfully completed jobs
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 20:36:14 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA_2974811c completed: *SUCCESS*
Experiment C96C48_hybatmDA_2974811c Completed at Tue Aug 22 22:51:10 UTC 2023
with 151 successfully completed jobs

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

TerrenceMcGuinness-NOAA commented Aug 22, 2023 •
edited
Loading

Yes, I can see those jobs were CANCELED by Slurm. Not sure what the root cause it yet. Could have been done by the admins for up-and-coming maintenance or for account allocations or limits. I'll keep you guys posted if I can find out.

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 20:27:08 UTC 2023
with 48 successfully completed jobs
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 20:36:14 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA_2974811c completed: *SUCCESS*
Experiment C96C48_hybatmDA_2974811c Completed at Tue Aug 22 22:51:10 UTC 2023
with 151 successfully completed jobs
Experiment C96_atm3DVar_2974811c completed: *SUCCESS*
Experiment C96_atm3DVar_2974811c Completed at Tue Aug 22 23:03:11 UTC 2023
with 89 successfully completed jobs

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added the CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully label Aug 22, 2023
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA removed the CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress label Aug 22, 2023
@TerrenceMcGuinness-NOAA
Copy link
Collaborator

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 20:27:08 UTC 2023
with 48 successfully completed jobs
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 20:36:14 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA_2974811c completed: *SUCCESS*
Experiment C96C48_hybatmDA_2974811c Completed at Tue Aug 22 22:51:10 UTC 2023
with 151 successfully completed jobs
Experiment C96_atm3DVar_2974811c completed: *SUCCESS*
Experiment C96_atm3DVar_2974811c Completed at Tue Aug 22 23:03:11 UTC 2023
with 89 successfully completed jobs

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

TerrenceMcGuinness-NOAA commented Aug 22, 2023 •
edited
Loading

Is this expected ...

mterry (Orion-login-1) 2021122100 $ grep FAILURE gdasanal.log | wc
571178 5140602 38268926
mterry (Orion-login-1) 2021122100 $ grep WARNING gdasanal.log | wc
97 684 7870
mterry (Orion-login-1) 2021122100 $ awk 'NR >= 575595 && NR <= 575600' gdasanal.log
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
75: READ_PREPBUFR: WARNING invalid pressure pob= 0.000000000000000E+000
75: at k= 1 for obstype=t kx= 180 c_station_id=
75: 0022958 reset pob= 1000000000.00000
mterry (Orion-login-1) 2021122100 $

@aerorahul aerorahul mentioned this pull request Aug 23, 2023
Closed
@aerorahul
Copy link
Contributor Author

Is this expected ...

mterry (Orion-login-1) 2021122100 $ grep FAILURE gdasanal.log | wc
571178 5140602 38268926
mterry (Orion-login-1) 2021122100 $ grep WARNING gdasanal.log | wc
97 684 7870
mterry (Orion-login-1) 2021122100 $ awk 'NR >= 575595 && NR <= 575600' gdasanal.log
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
75: READ_PREPBUFR: WARNING invalid pressure pob= 0.000000000000000E+000
75: at k= 1 for obstype=t kx= 180 c_station_id=
75: 0022958 reset pob= 1000000000.00000
mterry (Orion-login-1) 2021122100 $

I am not sure @TerrenceMcGuinness-NOAA
This is for the science team to verify if this output is expected. Either way, the runs on Hera completed without an error, so I am confident the Orion failures were machine related and these may not be relevant to the failure.

@CatherineThomas-NOAA
Copy link
Contributor

Thanks for setting up these tests @aerorahul. Can you or @TerrenceMcGuinness-NOAA point me to the full log for the Orion failure? I don't think it's related to this PR but I'd like to check out the antenna correction error.

@aerorahul
Copy link
Contributor Author

Failed logs on Orion at:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1818/RUNTESTS/COMROT/C96_atm3DVar_2974811c/logs/2021122100/gdasanal.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1818/RUNTESTS/COMROT/C96_atm3DVar_2974811c/logs/2021122100/gfsanal.log

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

Let's set off the Orion CI tests again since it's been a full day after the maintenance updates.

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Aug 25, 2023
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Aug 25, 2023
@emcbot
Copy link

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c

@emcbot
Copy link

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs

@emcbot
Copy link

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Fri Aug 25 10:32:09 CDT 2023
with 48 successfully completed jobs

@emcbot
Copy link

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Fri Aug 25 10:32:09 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Fri Aug 25 10:36:17 CDT 2023
with 3 successfully completed jobs

@emcbot
Copy link

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Fri Aug 25 10:32:09 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Fri Aug 25 10:36:17 CDT 2023
with 3 successfully completed jobs
Experiment C96_atm3DVar_2974811c completed: *SUCCESS*
Experiment C96_atm3DVar_2974811c Completed at Fri Aug 25 12:56:08 CDT 2023
with 89 successfully completed jobs

@emcbot
Copy link

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Fri Aug 25 10:32:09 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Fri Aug 25 10:36:17 CDT 2023
with 3 successfully completed jobs
Experiment C96_atm3DVar_2974811c completed: *SUCCESS*
Experiment C96_atm3DVar_2974811c Completed at Fri Aug 25 12:56:08 CDT 2023
with 89 successfully completed jobs
Experiment C96C48_hybatmDA_2974811c completed: *SUCCESS*
Experiment C96C48_hybatmDA_2974811c Completed at Fri Aug 25 13:06:09 CDT 2023
with 151 successfully completed jobs

@emcbot emcbot added CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Aug 25, 2023
@emcbot
Copy link

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Fri Aug 25 10:32:09 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Fri Aug 25 10:36:17 CDT 2023
with 3 successfully completed jobs
Experiment C96_atm3DVar_2974811c completed: *SUCCESS*
Experiment C96_atm3DVar_2974811c Completed at Fri Aug 25 12:56:08 CDT 2023
with 89 successfully completed jobs
Experiment C96C48_hybatmDA_2974811c completed: *SUCCESS*
Experiment C96C48_hybatmDA_2974811c Completed at Fri Aug 25 13:06:09 CDT 2023
with 151 successfully completed jobs

@aerorahul
Copy link
Contributor Author

Closing this PR as testing has concluded.

@aerorahul aerorahul closed this Aug 28, 2023
@CatherineThomas-NOAA
Copy link
Contributor

73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data

These errors in the Orion log are from a problem with the CRTM installation. As pointed out by @emilyhcliu, the MHS MetOp-C coefficient file is incorrect. It only contains header information:

Orion-login-4[4] cthomas$ pwd
/apps/contrib/NCEP/libs/hpc-stack/intel-2018.4/crtm/2.4.0/fix
Orion-login-4[5] cthomas$ ls -l mhs*
-rw-r----- 1 8536 noaa-hpc 11396 Feb  4  2022 mhs_metop-a.SpcCoeff.bin
-rw-r----- 1 8536 noaa-hpc 49212 Feb  4  2022 mhs_metop-a.TauCoeff.bin
-rw-r----- 1 8536 noaa-hpc 11396 Feb  4  2022 mhs_metop-b.SpcCoeff.bin
-rw-r----- 1 8536 noaa-hpc 49212 Feb  4  2022 mhs_metop-b.TauCoeff.bin
-rw-r----- 1 8536 noaa-hpc   492 Feb  4  2022 mhs_metop-c.SpcCoeff.bin
-rw-r----- 1 8536 noaa-hpc 49212 Feb  4  2022 mhs_metop-c.TauCoeff.bin

Which repo should I open an issue under to get this fixed?

@aerorahul
Copy link
Contributor Author

seems like an installation issue with CRTM.
hpc-stack seems relevant.

@CatherineThomas-NOAA CatherineThomas-NOAA mentioned this pull request Aug 31, 2023
Open
@aerorahul aerorahul deleted the feature/issue-1775 branch December 3, 2023 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@aerorahul @TerrenceMcGuinness-NOAA @emcbot @CatherineThomas-NOAA