Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a script to generate/edit output stream yaml files #2727

Merged
merged 18 commits into from
Mar 13, 2024

Conversation

bartgol
Copy link
Contributor

@bartgol bartgol commented Feb 22, 2024

This script allows to create and edit output yaml files via scripting. The main reason for this PR is to be able to create testmods that add/edit output streams for our nightly testing, avoiding the combinatorial explosion due to all the possible output stream options.

So far I added support for the basic options, but I still need to add the not-so-commonly-used ones, like fp precision or restart power-user options.

@bartgol bartgol added AT: AUTOMERGE Inform the autotester (AT) that it can merge this PR if reviewers approved, and tests pass AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested scripts AT: Skip weaver Do not run AT testing on weaver for this PR AT: Skip Stand-Alone Testing Do not run SCREAM internal unit tests for AT runs AT: Skip v1 Testing Do not run v1 CIME-based tests for AT runs Quality of Life labels Feb 22, 2024
@bartgol bartgol self-assigned this Feb 22, 2024
@bartgol bartgol removed AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested AT: Skip v1 Testing Do not run v1 CIME-based tests for AT runs labels Feb 23, 2024
@bartgol bartgol force-pushed the bartgol/eamxx/output-stream-edit-script branch from 228bf29 to 1731f89 Compare February 23, 2024 22:40
Copy link

github-actions bot commented Feb 23, 2024

PR Preview Action v1.4.7
🚀 Deployed preview to https://E3SM-Project.github.io/scream/pr-preview/pr-2727/
on branch gh-pages at 2024-03-13 17:41 UTC

@bartgol
Copy link
Contributor Author

bartgol commented Feb 23, 2024

@AaronDonahue @jgfouca This PR does quite a few changes, and I'm open to break it apart if you feel like it. Let me summarize them here

  • Add a script to generate/edit an output yaml file. It does not support all the options available (e.g., no support for file max snapshots, or for power-user restart options), but it is definitely enough for our testing purposes
  • Add a bunch of testmods, which create different output configurations. I basically have 3 streams: dyn, phys, diags. Other testmods can set average avg type, horiz, or vert remap. I added a bunch of fields to the streams, but probably could add more. @AaronDonahue , feel free to suggest more.
  • I added some "preset" testmods, which combine the testmods above, for a variety of output configurations. They are not exhaustive, but definitely an improvement w.r.t. what we had. Right now there are 6 presets. I don't think we should add more, cause we don't have enough tests to use more than those. If we think there has to be more stress-testing, we can make the presets more diverse, so that we test more configs with those 6 testmods.
  • I removed the default output yaml. By default, scream generates no output, unless told to do so. I find it clearer than just shoving all possible fields in a single file. Prod runs all use their own sets of yaml files anyways...
  • The above led me to a nano fix to atm_manip, when it comes to appending an option to an empty node.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4931
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;AT: Skip weaver;AT: Skip Stand-Alone Testing;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 1731f89
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 97d6e98
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: -1
  • Status: SKIPPED

Jenkins Parameters

Parameter Name Value

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/output-stream-edit-script
  • SHA: 1731f89
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Autotester
Copy link
Collaborator

NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester...

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4932
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;AT: Skip weaver;AT: Skip Stand-Alone Testing;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 1731f89
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 97d6e98
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: -1
  • Status: SKIPPED

Jenkins Parameters

Parameter Name Value

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/output-stream-edit-script
  • SHA: 1731f89
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Autotester
Copy link
Collaborator

NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester...

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4933
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;AT: Skip weaver;AT: Skip Stand-Alone Testing;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 1731f89
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 97d6e98
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: -1
  • Status: SKIPPED

Jenkins Parameters

Parameter Name Value

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/output-stream-edit-script
  • SHA: 1731f89
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Autotester
Copy link
Collaborator

NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester...

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4934
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;AT: Skip weaver;AT: Skip Stand-Alone Testing;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 1731f89
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 97d6e98
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: -1
  • Status: SKIPPED

Jenkins Parameters

Parameter Name Value

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/output-stream-edit-script
  • SHA: 1731f89
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Autotester
Copy link
Collaborator

NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester...

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4935
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;AT: Skip weaver;AT: Skip Stand-Alone Testing;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 1731f89
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 97d6e98
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: -1
  • Status: SKIPPED

Jenkins Parameters

Parameter Name Value

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/output-stream-edit-script
  • SHA: 1731f89
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Autotester
Copy link
Collaborator

NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester...

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4936
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;AT: Skip weaver;AT: Skip Stand-Alone Testing;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 1731f89
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 97d6e98
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: -1
  • Status: SKIPPED

Jenkins Parameters

Parameter Name Value

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/output-stream-edit-script
  • SHA: 1731f89
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Autotester
Copy link
Collaborator

NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester...

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 4937
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;AT: Skip weaver;AT: Skip Stand-Alone Testing;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 1731f89
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 97d6e98
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: -1
  • Status: SKIPPED

Jenkins Parameters

Parameter Name Value

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/output-stream-edit-script
  • SHA: 1731f89
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Autotester
Copy link
Collaborator

NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester...

Copy link
Member

@jgfouca jgfouca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great!

@AaronDonahue
Copy link
Contributor

@bartgol , I think this is ready to merge. Do you think it makes sense to rebase off of master since the last round of AT is about a week old?

@bartgol
Copy link
Contributor Author

bartgol commented Mar 13, 2024

@bartgol , I think this is ready to merge. Do you think it makes sense to rebase off of master since the last round of AT is about a week old?

Yeah, def needs a rebase and another round of testing.

When appending to an empty array, avoid leading ','
* Make some params case insensitive
* Add arg for filename prefix
…rs names

YAML_GEN and YAML_EDIT were both referring to the same script,
so use a single env var
Do not accept STOP_N/STOP_OPTION. These *do* change in an ERP/ERS test.
Besides, these refer to when the run should stop, not when the output
should be written.

Instead, use HIST_N/HIST_OPTION, which are the same across both runs
of an ERP/ERS test.
@bartgol bartgol force-pushed the bartgol/eamxx/output-stream-edit-script branch from 1a7ad5b to 9a0c6f8 Compare March 13, 2024 17:39
@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5038
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 9a0c6f8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 609bccb
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5413
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 9a0c6f8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 609bccb
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/output-stream-edit-script
  • SHA: 9a0c6f8
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Autotester
Copy link
Collaborator

NOTICE: The AutoTester has encountered an internal error (usually a Communications Timeout), testing will be restarted, previous tests may still be running but will be ignored by the AutoTester...

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5039
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 9a0c6f8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 609bccb
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5414
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 9a0c6f8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 609bccb
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/output-stream-edit-script
  • SHA: 9a0c6f8
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Autotester
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5039
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 9a0c6f8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 609bccb
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5414
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE;scripts;Quality of Life
PULLREQUESTNUM 2727
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 9a0c6f8
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 609bccb
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5039 FAILED (click to see last 100 lines of console output)

Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2--scream-output-preset-5 in 149.765156 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2--scream-output-preset-5 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2--scream-output-preset-5 in 0.855272 seconds (PEND). [COMPLETED 2 of 6]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.scream-output-preset-2 in 214.882289 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.scream-output-preset-2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.scream-output-preset-2 in 8.190067 seconds (PEND). [COMPLETED 3 of 6]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 in 239.697882 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 in 5.313749 seconds (PEND). [COMPLETED 4 of 6]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 in 392.081091 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 in 1.060544 seconds (PEND). [COMPLETED 5 of 6]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-4 in 437.680000 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-4 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-4 in 1.197681 seconds (PEND). [COMPLETED 6 of 6]
Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1.C.20240313_132607_h09cmz
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-4.C.20240313_132607_h09cmz
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2--scream-output-preset-5.C.20240313_132607_h09cmz
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.scream-output-preset-2.C.20240313_132607_h09cmz
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1.C.20240313_132607_h09cmz
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.scream-output-preset-3.C.20240313_132607_h09cmz
test-scheduler took 678.3807277679443 seconds'
+ errors='Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1.C.20240313_132607_h09cmz
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-4.C.20240313_132607_h09cmz
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2--scream-output-preset-5.C.20240313_132607_h09cmz
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.scream-output-preset-2.C.20240313_132607_h09cmz
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1.C.20240313_132607_h09cmz
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.scream-output-preset-3.C.20240313_132607_h09cmz
test-scheduler took 678.3807277679443 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1.C.20240313_132607_h09cmz
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-4.C.20240313_132607_h09cmz
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2--scream-output-preset-5.C.20240313_132607_h09cmz
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.scream-output-preset-2.C.20240313_132607_h09cmz
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1.C.20240313_132607_h09cmz
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.scream-output-preset-3.C.20240313_132607_h09cmz
test-scheduler took 678.3807277679443 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1.C.20240313_132607_h09cmz
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-4.C.20240313_132607_h09cmz
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-rad_frequency_2--scream-output-preset-5.C.20240313_132607_h09cmz
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.scream-output-preset-2.C.20240313_132607_h09cmz
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu9.scream-output-preset-1.C.20240313_132607_h09cmz
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu9.scream-output-preset-3.C.20240313_132607_h09cmz
test-scheduler took 678.3807277679443 seconds
  SCREAM SCRIPTS TESTING FAILED!
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 89936 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel

[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins7637840908214460147.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5414 PASSED (click to see last 100 lines of console output)

101/115 Test #101: create_vert_remap_and_weights ...........................   Passed    0.76 sec
        Start 102: shoc_p3_source
102/115 Test #102: shoc_p3_source ..........................................   Passed    2.49 sec
        Start 103: shoc_p3_nudged
103/115 Test #103: shoc_p3_nudged ..........................................   Passed    1.96 sec
        Start 104: shoc_p3_nudged_remapped
104/115 Test #104: shoc_p3_nudged_remapped .................................   Passed    1.81 sec
        Start 105: shoc_p3_nudging_glob_novert
105/115 Test #105: shoc_p3_nudging_glob_novert .............................   Passed    1.92 sec
        Start 106: homme_shoc_cld_p3_rrtmgp_np1
106/115 Test #106: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    4.25 sec
        Start 107: homme_shoc_cld_p3_rrtmgp_pg2_np1
107/115 Test #107: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    4.25 sec
        Start 108: model_baseline
108/115 Test #108: model_baseline ..........................................   Passed    5.21 sec
        Start 109: model_initial
109/115 Test #109: model_initial ...........................................   Passed    4.26 sec
        Start 110: model_restart
110/115 Test #110: model_restart ...........................................   Passed    5.02 sec
        Start 111: restarted_vs_monolithic_check_np1
111/115 Test #111: restarted_vs_monolithic_check_np1 .......................   Passed    0.08 sec
        Start 112: homme_shoc_cld_spa_p3_rrtmgp_np1
112/115 Test #112: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    5.73 sec
        Start 113: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
113/115 Test #113: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    5.61 sec
        Start 114: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
114/115 Test #114: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.07 sec
        Start 115: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
115/115 Test #115: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   18.76 sec

100% tests passed, 0 tests failed out of 110

Label Time Summary:
bfbhash = 0.97 secproc (1 test)
check = 0.74 sec
proc (1 test)
cld = 46.49 secproc (8 tests)
cld_fraction = 3.00 sec
proc (1 test)
cosp = 1.86 secproc (1 test)
diagnostics = 34.93 sec
proc (18 tests)
dp = 18.76 secproc (1 test)
driver = 71.74 sec
proc (18 tests)
dynamics = 47.44 secproc (9 tests)
fail = 82.64 sec
proc (4 tests)
io = 31.98 secproc (12 tests)
nudging = 5.92 sec
proc (2 tests)
p3 = 254.17 secproc (15 tests)
p3_sk = 175.50 sec
proc (2 tests)
pg2 = 4.25 secproc (1 test)
physics = 526.32 sec
proc (33 tests)
remap = 3.22 secproc (1 test)
rrtmgp = 68.87 sec
proc (14 tests)
shoc = 109.64 secproc (15 tests)
spa = 36.11 sec
proc (7 tests)
surface_coupling = 1.57 secproc (1 test)
tms = 4.25 sec
proc (1 test)

Total Test time (real) = 778.14 sec

The following tests did not run:
30 - physics_saturation_baseline_fake (Disabled)
38 - p3_baseline_f90_fake (Disabled)
39 - p3_baseline_cxx_fake (Disabled)
46 - shoc_baseline_f90_fake (Disabled)
47 - shoc_baseline_cxx_fake (Disabled)

Testing '''9a0c6f814da09e6abf7df609f092af6b51062a55''' for test '''release'''

RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/ctest-build/release

Testing '''9a0c6f814da09e6abf7df609f092af6b51062a55''' for test '''full_sp_debug'''

RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/ctest-build/full_sp_debug

Testing '''9a0c6f814da09e6abf7df609f092af6b51062a55''' for test '''full_debug'''

RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicxx -DCMAKE_C_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/ppc64le-pwr9-rhel8/tpls/openmpi/4.1.4/gcc/11.3.0/base/vu2aei6/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx/ctest-build/full_debug
OVERALL STATUS: PASS
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -p -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5414/scream/components/eamxx
Completed analysis on weaver'

  • [[ 0 != 0 ]]
  • [[ 1 == 0 ]]
  • [[ weaver == \m\a\p\p\y ]]
  • set +x
    Performing Post build task...
    Match found for : : True
    Logical operation result is TRUE
    Running script : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/

./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins13251625737252183422.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Finished: SUCCESS

@bartgol
Copy link
Contributor Author

bartgol commented Mar 13, 2024

Since Jim already approved before the rebsae and the only fails are diffs due to missing baselines, I'm going to merge this.

@bartgol bartgol merged commit 4f13c83 into master Mar 13, 2024
13 of 14 checks passed
@bartgol bartgol deleted the bartgol/eamxx/output-stream-edit-script branch March 13, 2024 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AT: AUTOMERGE Inform the autotester (AT) that it can merge this PR if reviewers approved, and tests pass Quality of Life scripts
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants