Merge pull request #2 from ACCESS-NRI/move-tests-from-access-om2-configs

Move tests from `access-om2-configs` repository

README.md

@@ -1,57 +1,69 @@
-# General Repository Template
+## Model Configuration Pytests
 
-A general template repository for default settings when creating new repositories.
+These pytests are used as part CI checks for model configurations.
 
-This repository uses the Apache-2.0 license. `COPYRIGHT.txt` contains a current copyright statement which should be included at the top of all files.
+The checksum pytests are used for reproducibility CI checks in the [ACCESS-NRI/reproducibility](https://github.com/ACCESS-NRI/reproducibility) repository. The quick configuration tests are used in 
+[ACCESS-NRI/access-om2-configs](https://github.com/ACCESS-NRI/access-om2-configs).
 
-When creating a new repository you [can use this repository as a template](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-repository-from-a-template), to automate the creation of the correct license and COPYRIGHT statement.
+Code from these tests is adapted from COSIMAS's ACCESS-OM2's [
+bit reproducibility tests](https://github.com/COSIMA/access-om2/blob/master/test/test_bit_reproducibility.py).
 
-## COPYRIGHT Header
+### How to run tests manually
 
-Best practice suggests adding a copyright statement at the top of every source code file, or text file where it is possible to add a copyright statement without interfering with the purpose of the file. The reasoning is if a file is separated from the repository in which it resides then it may not be possible to ascertain it's licensing, which may hamper re-use.
-
-Making this as short and concise as possible reduces the overhead in including such a copyright statement. To that end using [SPDX identifiers](https://spdx.dev/ids/) is simple, efficient, portable and machine-readable.
-
-### Examples
-
-An example, short, copyright statement is reproduced below, as it might appear in different coding languages. Copy and add to files as appropriate: 
-
-#### plaintext
-It is common to include copyright statements at the bottom of a text document or website page
-```text
-© 2022 ACCESS-NRI and contributors. See the top-level COPYRIGHT file for details. 
-SPDX-License-Identifier: Apache-2.0
+1. First clone the pytest code into a separate directory.
+```sh
+git clone https://github.com/ACCESS-NRI/model-config-tests/ test-code
 ```
 
-#### python
-For code it is more common to include the copyright in a comment at the top
-```python
-# Copyright 2022 ACCESS-NRI and contributors. See the top-level COPYRIGHT file for details.
-# SPDX-License-Identifier: Apache-2.0
+2. Checkout an experiment (in this case it is using an ACCESS-OM2 config)
+```sh
+git clone https://github.com/ACCESS-NRI/access-om2-configs/ <experiment>
+cd <experiment>
+git checkout <branch/tag>
 ```
 
-#### shell
-```bash
-# Copyright 2022 ACCESS-NRI and contributors. See the top-level COPYRIGHT file for details.
-# SPDX-License-Identifier: Apache-2.0
+3. Setup payu
+```sh
+module use /g/data/vk83/modules
+module load payu/1.1
 ```
 
-##### FORTRAN
-```fortran
-! Copyright 2022 ACCESS-NRI and contributors. See the top-level COPYRIGHT file for details.
-! SPDX-License-Identifier: Apache-2.0
+4. Run the pytests
+```sh
+pytest <path/to/test-code>
 ```
 
-#### C/C++ 
-```c
-// Copyright 2022 ACCESS-NRI and contributors. See the top-level COPYRIGHT file for details.
-// SPDX-License-Identifier: Apache-2.0
+### Pytest Options
+
+The output directory for pytests defaults to `/scratch/$PROJECT/$USER/test-model-repro` and contains the following sub-directories:
+- `control` - contains copies of the model configuration used for each experiment run in the tests.
+- `lab` - contains `payu` model output directories containing `work` and `archive` sub-directories.
+
+This output directory also contains files generated by pytests, including the `CHECKSUM` file which is used as part of reproducibility CI workflows.
+
+To specify a different folder for pytest outputs, use `--output-path` command flag, for example:
+
+```sh
+pytest <path/to/test-code> --output-path /some/other/path/for/output
 ```
 
-### Notes
+By default, the control directory, e.g. the model configuration to test, is the current working directory. This can be set similarly to above by using the 
+`--control-path` command flag.
+
+The path containing the checksum file to check against can also be set using
+`--checksum-path` command flag. The default is the `testing/checksum/CHECKSUM`
+file which is stored in the control directory.
 
-Note that the date is the first time the project is created. 
+To run only CI reproducibility checksum tests, use `-m checksum`, e.g.
 
-The date signifies the year from which the copyright notice applies. **NEVER** replace with a later year, only ever add later years or a year range. 
+```sh
+pytest <path/to/test-code> -m checksum
+```
+
+To run quick configuration tests, use the `config` marker. To additionally run
+ACCESS-OM2 specific quick configuration tests, use `access_om2` marker,
+e.g.:
 
-It is not necessary to include subsequent years in the copyright statement at all unless updates have been made at a later time, and even then it is largely discretionary: they are not necessary as copyright is contingent on the lifespan of copyright holder +50 years as per the [Berne Convention](https://en.wikipedia.org/wiki/Berne_Convention).
+```sh
+pytest <path/to/test-code> -m "config or access_om2"
+```

conftest.py

@@ -0,0 +1,108 @@
+# Copyright 2024 ACCESS-NRI and contributors. See the top-level COPYRIGHT file for details.
+# SPDX-License-Identifier: Apache-2.0
+
+import os
+import pytest
+from pathlib import Path
+
+import yaml
+from ruamel.yaml import YAML
+
+
+@pytest.fixture(scope="session")
+def output_path(request):
+    """Set the output path: This contains control and lab directories for each
+    test and test output files - e.g. CHECKSUMS
+    """
+    path = request.config.getoption('--output-path')
+    if path is None:
+        # Set default to /scratch/PROJECT/USER/test-model-repro/
+        project = os.environ.get('PROJECT')
+        user = os.environ.get('USER')
+        path = f'/scratch/{project}/{user}/test-model-repro'
+    return Path(path)
+
+
+@pytest.fixture(scope="session")
+def control_path(request):
+    """Set the path of the model configuration directory to test"""
+    path = request.config.getoption('--control-path')
+    if path is None:
+        # Set default to current working directory
+        path = Path.cwd()
+    return Path(path)
+
+
+@pytest.fixture(scope="session")
+def checksum_path(request, control_path):
+    """Set the path of the model configuration directory to test"""
+    path = request.config.getoption('--checksum-path')
+    if path is None:
+        # Set default to checksum stored on model configuration
+        path = control_path / 'testing' / 'checksum' / 'historical-3hr-checksum.json'
+    return Path(path)
+
+
+@pytest.fixture(scope="session")
+def metadata(control_path: Path):
+    """Read the metadata file in the control directory"""
+    metadata_path = control_path / 'metadata.yaml'
+    # Use ruamel.yaml as that is what is used to read metadata files in Payu
+    # It also errors out if there are duplicate keys in metadata
+    content = YAML().load(metadata_path)
+    return content
+
+
+@pytest.fixture(scope="session")
+def config(control_path: Path):
+    """Read the config file in the control directory"""
+    config_path = control_path / 'config.yaml'
+    with open(config_path) as f:
+        config_content = yaml.safe_load(f)
+    return config_content
+
+
+@pytest.fixture(scope="session")
+def target_branch(request):
+    """Set the target branch - i.e., the branch the configuration will be
+    merged into. This used is to infer configuration information, if the
+    configuration branches follow a common naming scheme (e.g. ACCESS-OM2)"""
+    return request.config.getoption('--target-branch')
+
+
+# Set up command line options and default for directory paths
+def pytest_addoption(parser):
+    """Attaches optional command line arguments"""
+    parser.addoption("--output-path",
+                     action="store",
+                     help="Specify the output directory path for test output")
+
+    parser.addoption("--control-path",
+                     action="store",
+                     help="Specify the model configuration path to test")
+
+    parser.addoption("--checksum-path",
+                     action="store",
+                     help="Specify the checksum file to compare against")
+
+    parser.addoption("--target-branch",
+                     action="store",
+                     help="Specify the target branch name")
+
+
+def pytest_configure(config):
+    config.addinivalue_line(
+        "markers", "slow: mark tests as slow (deselect with '-m \"not slow\"')"
+    )
+    config.addinivalue_line(
+        "markers", "test: mark tests as testing test functionality"
+    )
+    config.addinivalue_line(
+        "markers", "checksum: mark tests to run as part of reproducibility CI tests"
+    )
+    config.addinivalue_line(
+        "markers", "config: mark as configuration tests in quick QA CI checks"
+    )
+    config.addinivalue_line(
+        "markers", "access_om2: mark as access-om2 specific tests in quick QA CI checks"
+    )