Home

Shell script-based Regression Test: rt.sh

$ git clone https://github.com/ufs-community/ufs-weather-model DATM-MOM6-CICE6
$ cd DATM-MOM6-CICE6
$ git submodule update --init --recursive
$ cd tests_datm
$ vi rt.sh (The default account to use is nems. If you cannot use nems account,please change the "ACCNR" parameter, e.g., ACCNR=marine-cpu
$ rt.sh -f -k (This command will compile the code and run all 4 test cases and keep the run directory after the runs are done.)

detect_machine.sh: detect and assign machine name, set account (nems is default)
compile.sh: build a model using CMake
Run regression test
- run_test.sh: sets environment variables, run directory, etc., and calls rt_datm.sh
- rt_datm.sh: prepares a canned case in the run directory, and calls rt_utils functions
- rt_utils.sh: contains utility functions, e.g.,
  - submit_and_wait
  - check_results
  - rocoto_create_compile_task, rocoto_create_run_task, rocoto_run, rocoto_kill
  - ecflow_create_compile_task, ecflow_create_run_task, ecflow_run, ecflow_kill

Two levels to set simulation parameters
- default_vars.sh sets default values
- <test-name> overrides default values, adds test-specific parameters, e.g.,
  - IATM=1760, JATM=880, FHMAX=48, WLCLK=45
Set environment variables that are passed onto various template files in parm/
- input.mom6.nml.IN
- nems.configure.*.IN
- model_configure.IN
Specify configuration templates to use, e.g.,
- NEMS_CONFIGURE=”nems.configure.cmeps_lstep_cold_atm_ocn_ice.IN”

Set up input data, grid data, etc. by copying files from baseline directory to run directory
Baseline directory contains
- Sub-directories for input data (e.g., CICE_IC, MOM6_IC, and DATM)
- Sub-directories for previous run results (e.g., RT-Baselines_1step_cold_start_100_cmeps_CFSR)
Make sure directories and files exist in RTPWD

Baseline directory (RTPWD)
- Hera: /scratch1/NCEPDEV/nems/emc.nemspara/RT/DATM-MOM6-CICE6/develop-YYYYMMDD
- Orion: /work/noaa/nems/emc.nemspara/RT/DATM-MOM6-CICE6/develop-YYYYMMDD
Run directory root (RUNDIR_ROOT)
- Hera: RUNDIR_ROOT=/scratch1/NCEPDEV/stmp2/${USER}/S2S_RT/rt_$$
- Orion: RUNDIR_ROOT=/work/noaa/stmp/${USER}/test/${USER}/S2S_RT/rt_$$
- RUNDIR=${RUNDIR_ROOT}/${TEST_NAME}
New baseline directory (NEW_BASELINE)
- Hera: /scratch1/NCEPDEV/stmp4/${USER}/S2S_RT/REGRESSION_TEST_INTEL
- Orion: /work/noaa/stmp/${USER}/test/${USER}/S2S_RT/REGRESSION_TEST_INTEL

Triggered by COMPILE row in rt.conf
compile.sh is a simple wrapper around build.sh script located in DATM-MOM6-CICE6 directory.

If you make code changes that are not expected to change simulation results, you can run full regression tests afterward to demonstrate your changes do not break anything
Currently, there are 4 standard regression tests on Hera and Orion
In DATM-MOM6-CICE6/tests_datm/ directory, use any one of the following:
- $ ./rt.sh -f >output 2>&1 &
- $ ./rt.sh -f -e (use ecFlow)
- $ ./rt.sh -fr (use Rocoto)
- $ ./rt.sh -fek (use ecFlow, keep run directory for post-run diagnosis)

Create a file, say my_test.conf, with a single COMPILE and a single RUN
- $ cp rt.conf my_test.conf
- $ vi my_test.conf
- $ ./rt.sh -l my_test.conf
Or make a copy of original rt.conf file
- $ cp rt.conf rt.conf.orig
- $ vi rt.conf
- $ ./rt.sh -f

Remove COMPILE row in rt.conf
cp DATM-MOM6-CICE6/modulefiles/hera.intel/coupled DATM-MOM6-CICE6/tests_datm/modules.fcst (on hera)
or cp DATM-MOM6-CICE6/modulefiles/orion.intel/coupled DATM-MOM6-CICE6/tests_datm/modules.fcst (on orion)
$ ./rt.sh -f
This approach does not work with workflow managers because RUN depends on COMPILE

Summary files
- Hera: RegressionTests_hera.intel.log, Compile_hera.intel.log
- Orion: RegressionTests_orion.intel.log, Compile_orion.intel.log
./rt.sh >output 2>&1 &: output of rt.sh
Log files in log_hera.intel/ and log_orion.intel/
- compile_*.log: output of compile.sh
- run_*.log: output of run_test.sh
Run directory RUNDIR_ROOT/
- .log: output of rt_datm.sh. If rocoto used, also contains err & out from sbatch job
- subdir: contains all files necessary for simulation, e.g., sbatch job_card
- run_test*.env: exported environment variables, e.g., MACHINE_ID, RTPWD, PATHRT, ACCNR, QUEUE.