This is the Regression Testing Suite (RTS) for the WRF Data Assimilation system (WRFDA).

0. Purpose

  The purpose of this testing system is to prevent regressions, or the introduction of software 
  bugs, to the WRFDA system. The WRFDA RTS achieves this by running different tests on components 
  of the WRFDA system, the comparing the output to a known baseline. The tests are designed to be 
  customizable, compartmentalized, and easy to create.

               IT IS IMPORTANT TO NOTE THAT THIS SYSTEM CAN NOT DETECT INCORRECT RESULTS

  This testing suite can only ensure that all files that should have been created are created, and 
  that the output of the test has not changed since the version which was used to generate the 
  baseline. There are also sanity checks for NaNs in the output and cases where the first guess 
  file is bit-for-bit identical to the output, but poor assimilation can not be detected.

1. Contents 

  The contents of the WRFDA regression testing suite are as follows. Some files and directories 
  are required for proper functioning, while others are optional but helpful additions.

  Required files and directories:

  regtest.pl      The Perl script which runs the regression test suite
  testdata.txt    The text file which holds the test information for each platform
  WRFDA-data-EM/  The directory containing data used to run each test. Can specify different name 
                  in 'testdata.txt'
  BASELINE.NEW/   The directory containing baseline comparisons for each test. Can specify 
                  different name in 'testdata.txt'
  wrfda.tar       The WRFDA source code you wish to use. The name can be specified in 
                  'textdata.txt' or the code can be retrieved from a Subversion or Git repository.

  Optional files and directories:

  scripts/        Contains helpful scripts, such as 'quickcopy.pl' which copies test results to be 
                  used as the new baseline.
  README          This README file

2. Supported platforms

  This regression testing suite is optimized for use on the NCAR Yellowstone supercomputer. 
  However, with minimal effort it can run (albeit slowly) on personal computers, both Linux and 
  Mac.

  Supported Fortran compilers are ifort, gfortran, and pgf90. Other compilers which should work 
  with minimal effort are xlf and g95.


3. Set up

  The WRFDA Regression Testing Suite is kept in the following Git repository branch (contact
  [email protected] or Michael Kavulich; [email protected]) for access:

    https://github.com/wrf-model/WRFDA_REGTEST

  In this repository you will find 
    - the main testing script (regtest.pl)
    - a text file containing test information (testdata.txt)
    - a directory containing a few helpful scripts (scripts/)
    - a text file containing data about compile times (compile_baseline.txt)
    - this README file

  In addition to the files found in the above repository, you must download two data sets for the 
  regression testing to function:

    - The test database (WRFDA-data-EM), containing the necessary files for each test to run
    - The baseline output files for comparison (BASELINE.NEW)

  These data sets can be found at the WRFDA website: 
  http://www.mmm.ucar.edu/wrf/users/wrfda/regression/index.html

  This system is not yet set up to automatically compile WRFPLUS for 4DVAR tests. Therefore, to 
  run 4DVAR tests, you must compile the appropriate code separately. The testing suite will look 
  for the WRFPLUS code in the test's 'libs' directory, in a sub-directory named 
  'WRFPLUSV3_compiler', where "compiler" is gnu, intel, pgi, etc. For serial 4DVAR tests, you'll 
  need to compile WRFPLUS for serial runs as well, and the testing suite will look for the WRFPLUS 
  code in a sub-directory of 'libs' named 'WRFPLUSV3_compiler_serial'.

  Certain tests require external libraries, such as RTTOV and HDF5. You should link/copy the full 
  installation of these libraries into the provided blank 'libs/' directory. These directories 
  should follow the same compiler-specific naming conventions as WRFPLUS; for example, RTTOV 
  compiled with gfortran should be found in the directory 'libs/RTTOV_gfortran', and HDF5 
  libraries in the directory 'libs/HDF5_gfortran'. These directories should contain the full 
  library installation, so the command `ls libs/HDF5_gfortran` in the above example should show 
  the following subdirectories:

     bin     include lib     share

4. Running WRFDA RTS

  The RTS can be run by executing the main script: 'regtest.pl'

    > ./regtest.pl

  However, without any input arguments, the script will fail with the following helpful message:

A compiler must be specified!

Aborting the script with dignity.

Usage : regtest.pl --compiler=COMPILER --cc=C_COMPILER --source=SOURCE_CODE.tar --revision=NNNN --upload=[no]/yes
                   --j=NUM_PROCS --exec=[no]/yes --debug=[no]/yes/super --testfile=testdata.txt
                   --netcdf4=[no]/yes --hdf5=no/[yes] --rttov=no/[yes] --cloudcv=[no]/yes
                   [email protected]:wrf-model/WRF --fork=github_username --branch=branchname
                   --tests='testname1 testname2' --libs=`pwd`/libs

        compiler: [REQUIRED] Compiler name (supported options: ifort, gfortran, xlf, pgf90, g95)
        cc:       C Compiler name (supported options: icc, gcc, xlf, pgcc, g95)
        source:   Specify location of source code in one of 3 formats:
                   - If a path is specified, script will assume that path contains the WRFDA source code
                   - If a .tar or .gz file is specified, that archive should contain a directory named 'WRFDA'
                   - If 'REPO' is specified, script will look for code from a remote repository; see 'repo' option
        revision: Specify code revision to retrieve (only works when '--source=REPO' specified)
                  Use any number to specify that revision number; specify 'HEAD' to use the latest revision
        upload:   Uploads summary to web (default is 'yes' iff source==REPO and revision==HEAD)
        j:        Number of processors to use in parallel compile (default 4, use 1 for serial compilation)
        exec:     Execute only; skips compile, utilizes existing executables
        debug:    'yes' compiles with minimal optimization; 'super' compiles with debugging options as well
        cloudcv:  Compile for CLOUD_CV options
        netcdf4:  Compile for NETCDF4 options
        hdf5:     Compile for HDF5 options (note that the default value is 'yes')
        rttov:    Compile for RTTOV options (note that the default value is 'yes')
        testfile: Name of test data file (default: testdata.txt)
        repo:     Location of code repository
        fork:     (git only) Overwrites default repository '[email protected]:wrf-model/WRF' to '[email protected]:fork/WRF'
        branch:   (git only) branch of repository to use
        tests:    Test names to run (prunes test list taken from testfile; test specs must exist there!)
        libs:     Specify where the necessary libraries are located

  As should be obvious from the above message, there are a number of command-line options you can 
  provide to the script to run the RTS in different ways. Let's examine these options in detail:

  compiler:
    This is the name of the Fortran compiler you will be using to compile WRFDA. Active testing 
    takes place with gfortran, ifort, and pgf90, but you should be able to easily adapt this test 
    to use any Fortran compiler that WRF is set up for. This is the only mandatory command-line 
    argument for regtest.pl
  source:
    This is the source of the code you will be testing. You can enter one of three options:
      --source=REPO will check out a version of the code from the WRF code repository at 
                   https://svn-wrf-model.cgd.ucar.edu/trunk/ (or specified by the "repo" option). 
                   By default, the most recent (HEAD) repository will be checked out, but you can 
                   use the "revision" option to change this
      --source=SOURCE_CODE.tar (or .gz) will use a tar (gzip) file that you specify which contains
                   the WRFDA source code. It must be a tar file which only contains the WRFDA code
                   in a directory named "WRFDA". You can specify a tar file in a different local 
                   directory with either a full path or a relative path.
      --source=path-to-code will look for the WRFDA code in the specified path
  revision:
    If you specify "--source=REPO", you can use "--revision=####" to specify a specific revision 
    number (for SVN) or hash (for Git) to check out of the repository
  repo:
    The default repository when you specify "--source=REPO" is 
    "[email protected]:wrf-model/WRF". You can specify a different repository with this option.
  fork:
    For Git repositories only. The default repostory when you specify "--source=REPO" is
    "[email protected]:wrf-model/WRF". If you do not specify a "--repo" option, you can specify a fork
    which will get the repository located at [email protected]:"fork"/WRF. In almost all cases "fork"
    should be a github username.
  branch:
    For Git repositories only. The default branch when you specify "--source=REPO" is "master". You
    can specify a different branch with this option.
  upload:
    Will upload a summary of the test to the web (found at 
    http://www2.mmm.ucar.edu/wrf/users/wrfda/regression/index.html). You must have write permission
    on the MMM "nebula" machine to successfully use this option, as you will be prompted for your 
    password to upload the summary files
  exec:
    If you have run a previous test and do not wish to delete that code and re-compile, you can use
    "--exec=yes" to re-run the tests on existing code
  debug:
    Allows you to compile WRFDA without optimization (--debug=yes) or with debug flags and no 
    optimization (--debug=super). Results may differ from the baseline generated without these 
    options, but debug options can help trap errors not caught by the standard tests. 
  j:
    Specify the number of cores to be used with parallel make to speed up compilation. The default 
    is 4, and 16 is the maximum possible value, though values greater than 6 are not recommended, 
    as there is minimal speedup beyond this
  cloudcv:
    Compiles WRFDA with the "CLOUDCV=1" environment variable set, which compiles a lot of if-deffed
    code for moisture control variables that is normally not compiled. There are currently no tests
    specifically designed for this option, and it will cause CV3 tests to fail
  netcdf4:
    Compile for NETCDF4 options.
  hdf5:
    Compile using HDF5 libraries. On by default if the libraries are found in 'libs/HDF5_compiler',
    but setting this to --hdf5=no will disable this option. Required for amsr2 test(s).
  testfile:
    You can read a different test file than the default 'testdata.txt' with this option.
  tests:
    If you wish to run just a subset of tests read from the "testfile" specified above without
    modifying that file, you can give a list of test names as a space-separated string, like this:
    "--tests='test_name_1 test_name_2 test_name_3'. The specified test names MUST appear in the 
    testfile.
  libs:
    Specify where the necessary libraries are located. Default is "libs/", a subdirectory of the 
    path from which the script is run. An absolute path may also be specified.

5. Output

  Most of the messages found in the regression test summary are self-explanatory. Here's a full 
  list of possible outcomes:

  If the test completed successfully:
  ------------------------------------------------------------------------------------------------
  match  The output of the test is a bit-for-bit match with the baseline. Excellent news!
  ok     The output of the test is not a bit-for-bit match with the baseline, but the fields are 
         exactly the same. This can result from a change in the variables stored in output, for 
         instance. This is a perfectly acceptable result, but you should update the baseline using 
         the quickcopy.pl script. NOTE: Tests using NETCDF4 files will always report 'ok' due to 
         limitations of the testing routines that have not yet been resolved.
  diff   This means that the output of the test is different than the baseline. This often means 
         there is a problem, though if you expect your updates to change the output this may be an 
         acceptable result. Check the output thoroughly!

  If the test failed:
  ------------------------------------------------------------------------------------------------
  obsproc failed         For obsproc tests, this status message means that obsproc did not create 
                         an observation file.
  genbe failed           For gen_be tests, this status message means that gen_be reported failure.
  Output missing         wrfvar_output was not created. Check the rsl.* files in the test
                         directory.
  Baseline missing       The test could not find a baseline file to compare to your test results.
  Unknown error          Something strange went wrong in the baseline comparison subroutine.
  Mysterious error!      Something strange went wrong (the subroutine for starting the test 
                         returned "undef"). 
  diffwrf                diffwrf (used for comparing your output to the baseline) failed for some 
     comparison failed   reason (ensure that it is installed on your system and in your $PATH)
  Could not open output  Your output or baseline file is malformed (or possibly missing).
         and/or baseline
  fg == wrfvar_output    The test produced output, but the output is equal to the first guess, 
                         likely indicating a problem.