-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update EAMxx docs and improve help formatters #3044
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,17 @@ | ||
# Continuous Integration and Nightly Testing | ||
|
||
* Autotester quick overview | ||
* Nightly overview, CDash | ||
## Autotester ## | ||
|
||
EAMxx using github actions and a Sandia product called Autotester 2 | ||
to run CI testing on a CPU and GPU machine for every github pull | ||
request. By default, we run the e3sm_scream_v1_at suite and the | ||
standalone eamxx tests (test-all-scream). | ||
|
||
## Nightly overview, CDash ## | ||
|
||
Our nightly testing is much more extensive than the CI testing. You | ||
can see our dashboard here under the section "E3SM_SCREAM": | ||
https://my.cdash.org/index.php?project=E3SM | ||
|
||
We run a variety of CIME test suites and standalone testing on a number | ||
of platforms. We even do some performance testing on frontier. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,59 @@ | ||
# Full Model Testing | ||
|
||
Quickly review CIME test infrastructure and how EAMxx uses it | ||
Full model system testing of eamxx is done through CIME test cases | ||
(much like the rest of E3SM). | ||
|
||
* test types, specifiers (`_LnX`,`_D`,`_PMxN`,..), grids, compsets, test-mods | ||
* available grids/compsets for EAMxx, and where to find them | ||
* how to add atmchange in `shell_commands` test mods | ||
We offer a number of test suites, including: | ||
* e3sm_scream_v0: Test the full set of V0 (pre-C++) tests | ||
* e3sm_scream_v1: Test the full set of V1 (C++) tests | ||
* e3sm_scream_v1_at: A smaller and quicker set of tests for autotesting | ||
* e3sm_scream_hires: A small number of bigger, longer-running tests to measure performance | ||
|
||
Example for running a suite: | ||
``` | ||
% cd $repo/cime/scripts | ||
% ./create_test e3sm_scream_v1_at --wait | ||
``` | ||
|
||
Example for running a single test case: | ||
``` | ||
% cd $repo/cime/scripts | ||
% ./create_test SMS.ne4_ne4.F2010-SCREAMv1 --wait | ||
``` | ||
|
||
There are many behavioral tweaks you can make to a test case, like | ||
changing the run length, test type, etc. Most of this is not specific | ||
to eamxx and works for any CIME case. This generic stuff | ||
is well-documentated here: | ||
http://esmci.github.io/cime/versions/master/html/users_guide/testing.html | ||
|
||
When it comes to things specific to eamxx, you have grids, compsets, and | ||
testmods. | ||
|
||
Common EAMxx grids are: | ||
* ne4_ne4 (low resolution) | ||
* ne4pg2_ne4pg2 (low resolution with phys grid) | ||
* ne30_ne30 (med resolution) | ||
* ne30pg2_ne30pg2 (med resolution with phys grid) | ||
* ne1024pg2_ne1024pg2 (ultra high with phys grid) | ||
|
||
More grid info can be found here: | ||
https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/933986549/ATM+Grid+Resolution+Summary | ||
|
||
Common EAMxx compsets are: | ||
* F2010-SCREAM-LR: V0 low res compset with eamxx V0 atmosphere | ||
* F2010-SCREAMv1: V1 standard compset with eamxx V1 atmosphere | ||
* FIOP-SCREAMv1-DP: V1 with dpxx (doubly-periodic lateral boundary condition in C++) | ||
* F2010-SCREAMv1-noAero: V1 without aerosol forcing | ||
|
||
Full info on supported compsets can be found by looking at this file: | ||
`$scream_repo/components/eamxx/cime_config/config_compsets.xml` | ||
|
||
Common EAMxx testmods are: | ||
* small_kernels: Enable smaller-granularity kernels, can improve performance on some systems | ||
* scream-output-preset-[1-6]: Our 6 output presets. These turn some combination of our three output streams (phys_dyn, phys, and diags), | ||
various remaps, etc. | ||
* bfbhash: Turns on bit-for-bit hash output: https://acme-climate.atlassian.net/wiki/spaces/NGDNA/pages/3831923056/EAMxx+BFB+hashing | ||
|
||
More info on running EAMxx can be found here: | ||
https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3386015745/How+To+Run+EAMxx+SCREAMv1 |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -21,7 +21,43 @@ configurations. We use several types of tests | |
the [ctest](https://cmake.org/cmake/help/latest/manual/ctest.1.html) command. | ||
|
||
We also support a `test-all-scream` configuration that runs all of the | ||
standalone tests for an EAMxx configuration. | ||
standalone tests for an EAMxx configuration. Note, your current machine | ||
must be known to EAMxx before `test-all-scream` will work. A machine can | ||
be made known to EAMxx by editing the eamxx/scripts/machines_specs.py files. | ||
There are some instructions on what to do at the top of this file. | ||
|
||
`test-all-scream` has a good help dump | ||
``` | ||
% cd $scream_repo/components/eamxx | ||
% ./scripts/test-all-scream -h | ||
``` | ||
|
||
If you are unsure of the cmake configuration for you development cycle, one | ||
trick you can use is to run `test-all-scream` for the `dbg` test and just | ||
copy the cmake command it prints (then ctrl-C the process). | ||
``` | ||
% cd $scream_repo/components/eamxx | ||
% ./scripts/test-all-scream -t dbg -m $machine | ||
* wait for a few seconds* | ||
* Ctrl-C * | ||
* Copy the contents of DCMAKE_COMMAND that was passed to ctest * | ||
* Add "cmake" to beginning of contents and path to eamxx at the end. * | ||
``` | ||
|
||
Considerations for using `test-all-scream`: | ||
* Your machine must be known to our scripts, see above. | ||
* If you try to run commands by-hand (outside of test-all-scream; cmake, make, ctest, etc), you'll need to remember to | ||
load the scream-env into your shell, which can be done like this: | ||
`cd eamxx/scripts; eval $(./scripts/scream-env-cmd $machine)` | ||
* test-all-scream expects to be run from a compute node if you | ||
are on a batch machine. | ||
* You'll need to think about your baseline situation, as many of our | ||
tests rely on pre-existing baselines. The -b option controls the baseline | ||
location and can have the following values: | ||
* AUTO: A common public baseline area shared by all developers | ||
* LOCAL: A private baseline area for the current developer in the current repo | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Side question: what do you think about removing this? I don't think we ever use this feature, and it's just more code to maintain and code path to test in the scripts... The user can simply use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it's a nice little convenience option. I looked and it's literally 3 lines of code to support:
|
||
* $path: A specific arbitrary path | ||
* None: If there is no -b at all, no baseline testing will be done | ||
|
||
## Running EAMxx's Tests with CTest | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could mention the
--config-only
flag, rather than ctrl-C.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should also mention that they may have to run
eval $(./scripts/scream-env-cmd $mach)
before running make in the build folder...There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I forgot about this option. I just tried it and unfortunately you still have to wait over a minute:
I will make note of the need to load the scream env.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, cause it has to actually run cmake, while with ctrl-C you kill it after it shows the cmake line. That makes sense.
That said, we often need to build/run just one exec. For that, we need to eventually run cmake anyways, so we should advertise the
--config-only
option, I think.