Tier-2 tests

[guillaumevernieres]

Tier-2 tests would not be part of the regular GDASApp CI and would only be executed prior to submitting a PR of dev/gdasapp into develop.

Untested.

@CoryMartin-NOAA , @RussTreadon-NOAA , could you give me a sequential list of the jjobs that you want tested? That or commit directly to my branch. What needs to be updated is here

GDASApp/test/gw-ci/CMakeLists.txt

Line 61 in 6f27a49

set(TASK_LIST) # empty list for now

and here

GDASApp/test/gw-ci/CMakeLists.txt

Line 68 in 6f27a49

set(TASK_LIST) # empty list for now

and the marine DA example is here:

GDASApp/test/gw-ci/CMakeLists.txt

Line 47 in 6f27a49

set(TASK_LIST

[RussTreadon-NOAA]

g-w C96C48_ufs_hybatmDA CI runs the following jobs

• 20240223 18Z half cycle
  • gdasfcst and gdasatmos_prod_f*
  • enkfgdasfcst_mem[001, 002] and enkfgdasepos*
• 20240224 00Z full cycle - g-w CI also run gfs jobs but these can be omitted for GDASApp CI
  • gdasprep
  • gdasprepatmiodaobs
  • gdasatmanlinit
  • gdasatmanlvar
  • gdasatmanlfv3inc
  • gdasatmanlfinal
  • enkfgdasatmensanlinit
  • enkfgdasatmensanlletkf
  • enkfgdasatmensanlfv3inc
  • enkfgdasatmensanlfinal

g-w CI runs jobs after the above DA jobs finish. This is useful since it ensures the generated analysis increment files are readable by downstream jobs. However, adding these downstream jobs to GDASApp CI is too much at this point.

[CoryMartin-NOAA]

I probably won't be able to get you a list of aero+snow jobs until early next week, so don't wait for me if you want to proceed with this PR, otherwise I can get that to you Mon or Tues

[guillaumevernieres]

I updated the non-existent PR description @RussTreadon-NOAA , but the idea is to not run these as part of the regular GDASApp CI, but rather only prior to doing a PR in the global-workflow develop.

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Mon Jul  8 16:08:16 UTC 2024 on hfe12
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Jul  8 16:55:10 UTC 2024
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Mon Jul  8 17:18:34 UTC 2024
Tests: 100% tests passed, 0 tests failed out of 48

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Mon Jul  8 17:37:01 UTC 2024 on hfe06
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Jul  8 18:27:43 UTC 2024
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Mon Jul  8 18:51:07 UTC 2024
Tests: 100% tests passed, 0 tests failed out of 47
Tests: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1185/global-workflow/sorc/gdas.cd/build/log.ctest

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Mon Jul  8 23:43:57 UTC 2024 on hfe03
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Jul  9 00:29:42 UTC 2024
---------------------------------------------------
Tier-2 Testing: Activated
Tests:                                 *SUCCESS*
Tests: Completed at Tue Jul  9 00:58:31 UTC 2024
Tests: 100% tests passed, 0 tests failed out of 47

[guillaumevernieres]

These are the new tests added:

  Test #25: WCDA-3DVAR-C48mx500
  Test #26: WCDA-3DVAR-C48mx500_gdasfcst_202103241200
  Test #27: WCDA-3DVAR-C48mx500_enkfgdasfcst_001_202103241200
  Test #28: WCDA-3DVAR-C48mx500_enkfgdasfcst_000_202103241200
  Test #29: WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800
  Test #30: WCDA-3DVAR-C48mx500_gdasocnanalprep_202103241800
  Test #31: WCDA-3DVAR-C48mx500_gdasocnanalbmat_202103241800
  Test #32: WCDA-3DVAR-C48mx500_gdasocnanalrun_202103241800
  Test #33: WCDA-3DVAR-C48mx500_gdasocnanalchkpt_202103241800
  Test #34: WCDA-3DVAR-C48mx500_gdasocnanalpost_202103241800
  Test #35: Aero-Snow-3DVAR-C96
  Test #36: Aero-Snow-3DVAR-C96_gdasfcst_202112201200
  Test #37: Aero-Snow-3DVAR-C96_enkfgdasfcst_001_202112201200
  Test #38: Aero-Snow-3DVAR-C96_enkfgdasfcst_000_202112201200
  Test #39: Atm-hyb-C96C48
  Test #40: Atm-hyb-C96C48_gdasfcst_202402231800
  Test #41: Atm-hyb-C96C48_enkfgdasfcst_001_202402231800
  Test #42: Atm-hyb-C96C48_enkfgdasfcst_002_202402231800
  Test #43: Atm-hyb-C96C48_gdasprep_202402240000
  Test #44: Atm-hyb-C96C48_gdasprepatmiodaobs_202402240000
  Test #45: Atm-hyb-C96C48_gdasatmanlinit_202402240000
  Test #46: Atm-hyb-C96C48_gdasatmanlvar_202402240000
  Test #47: Atm-hyb-C96C48_gdasatmanlfv3inc_202402240000
  Test #48: Atm-hyb-C96C48_gdasatmanlfinal_202402240000
  Test #49: Atm-hyb-C96C48_enkfgdasatmensanlinit_202402240000
  Test #50: Atm-hyb-C96C48_enkfgdasatmensanlletkf_202402240000
  Test #51: Atm-hyb-C96C48_enkfgdasatmensanlfv3inc_202402240000
  Test #52: Atm-hyb-C96C48_enkfgdasatmensanlfinal_202402240000
  Test #53: GFSv17-3DVAR-C384mx025
  Test #54: GFSv17-3DVAR-C384mx025_gdasfcst_202106300000
  Test #55: GFSv17-3DVAR-C384mx025_enkfgdasfcst_001_202106300000
  Test #56: GFSv17-3DVAR-C384mx025_enkfgdasfcst_000_202106300000
  Test #57: GFSv17-3DVAR-C384mx025_gdasprepoceanobs_202106300600
  Test #58: GFSv17-3DVAR-C384mx025_gdasocnanalprep_202106300600
  Test #59: GFSv17-3DVAR-C384mx025_gdasocnanalbmat_202106300600
  Test #60: GFSv17-3DVAR-C384mx025_gdasocnanalrun_202106300600
  Test #61: GFSv17-3DVAR-C384mx025_gdasocnanalchkpt_202106300600
  Test #62: GFSv17-3DVAR-C384mx025_gdasocnanalpost_202106300600
  Test #63: GFSv17-3DVAR-C384mx025_gdasocnanalvrfy_202106300600
  Test #64: GFSv17-3DVAR-C384mx025_gdasprep_202106300600
  Test #65: GFSv17-3DVAR-C384mx025_gdasanal_202106300600
  [...]

Total Tests: 77

... and as I'm seeing that list, I'm thinking we probably need a new label or a way to trigger only a subset of these.

[RussTreadon-NOAA]

While I see Tier-2 testing was a SUCCESS , we don't see a hera-GW-RT-tier2-Passed label.

A check of labels only shows hera-GW-RT-tier2. Do we need to add Running, Failed, Passed labels for hera-GW-RT-tier2?

[guillaumevernieres]

While I see Tier-2 testing was a SUCCESS , we don't see a hera-GW-RT-tier2-Passed label.

A check of labels only shows hera-GW-RT-tier2. Do we need to add Running, Failed, Passed labels for hera-GW-RT-tier2?

Sorry for the confusion @RussTreadon-NOAA . The tier-2 test actually did not run, but even if they did, only the GW-RT label would be updated. The label is just there to trigger the tier-2 test, I could rename it to tier-2 only to make it less confusing?

[RussTreadon-NOAA]

Thanks @guillaumevernieres for the explanation. I think renaming the tier-2 test trigger label would be helpful. Would the tier-2 label trigger jobs on only Hera or would it also trigger jobs on Orion?

[guillaumevernieres]

Thanks @guillaumevernieres for the explanation. I think renaming the tier-2 test trigger label would be helpful. Would the tier-2 label trigger jobs on only Hera or would it also trigger jobs on Orion?

You could trigger them on other machines, but currently the marine DA ones would fail. We've been meaning to update the yamls in the g-w ... On our todo list.

[RussTreadon-NOAA]

Got it. The hera-GW-RT-tier2 currently works as the trigger on Hera. Given this, we can keep things simple, retain this label, and revisit (if needed) later.

[RussTreadon-NOAA]

Approve.

[guillaumevernieres]

Not sure what I did but the tier-2 is not triggered anymore ... Great.
I'll do some debugging while working on the g-w b-matrix pr.

[CoryMartin-NOAA]

@guillaumevernieres should we keep this PR alive or close it?