Changed osu test location.

.github/workflows/pip_install.yml

@@ -0,0 +1,33 @@
+# documentation: https://help.github.com/en/articles/workflow-syntax-for-github-actions
+name: Test installation of EESSI test suite with 'pip install'
+on: [push, pull_request, workflow_dispatch]
+permissions: read-all
+jobs:
+  test_pip_install:
+    runs-on: ubuntu-22.04
+    strategy:
+      fail-fast: false
+      matrix:
+        python: ['3.6', '3.7', '3.8', '3.9', '3.10', '3.11']
+    steps:
+        - name: Check out software-layer repository
+          uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0
+          with:
+            persist-credentials: false
+
+        - name: Install ReFrame
+          run: |
+            pip install --user ReFrame-HPC
+
+        - name: Install EESSI test suite with 'pip install'
+          run: |
+            # install from source distribution tarball, to test release as published on PyPI
+            rm -rf dist
+            python setup.py sdist
+            ls dist
+
+            pip install --user dist/eessi*.tar.gz
+            find $HOME/.local
+
+            python -c 'import eessi.testsuite.utils'
+            python -c 'import eessi.testsuite.tests.apps'

.github/workflows/test.yml

@@ -0,0 +1,64 @@
+# documentation: https://help.github.com/en/articles/workflow-syntax-for-github-actions
+name: Tests for EESSI test suite, using EESSI pilot repo
+on: [push, pull_request, workflow_dispatch]
+permissions: read-all
+jobs:
+  test_with_eessi_pilot:
+    runs-on: ubuntu-22.04
+    strategy:
+      fail-fast: false
+      matrix:
+        EESSI_VERSION:
+        - "2021.12"
+    steps:
+        - name: Check out software-layer repository
+          uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0
+          with:
+            persist-credentials: false
+
+        - name: Mount EESSI CernVM-FS pilot repository
+          uses: eessi/github-action-eessi@58b50fd2eead2162c2b9ac258d4fb60cc9f30503  # v2.0.13
+          with:
+            eessi_stack_version: ${{matrix.EESSI_VERSION}}
+
+        - name: Run test suite
+          run: |
+            # install latest version of EasyBuild, to install ReFrame with it,
+            # since that includes the ReFrame test library (hpctestlib) that we rely on
+            pip3 install --user easybuild
+            export PATH=$HOME/.local/bin:$PATH
+            eb --version
+            export EASYBUILD_PREFIX=$HOME/easybuild
+            # need to force module generation with --module-only --force because 'pip check' fails
+            # in EESSI pilot 2021.12, see https://github.com/EESSI/compatibility-layer/issues/152
+            eb ReFrame-4.2.0.eb || eb ReFrame-4.2.0.eb --module-only --force
+
+            # load ReFrame
+            module use $HOME/easybuild/modules/all
+            module load ReFrame/4.2.0
+            reframe --version
+
+            # configure ReFrame (cfr. https://reframe-hpc.readthedocs.io/en/stable/manpage.html#environment)
+            export RFM_CONFIG_FILES=$PWD/config/github_actions.py
+            export RFM_SYSTEM=github_actions_eessi
+            export RFM_CHECK_SEARCH_PATH=$PWD/eessi/testsuite/tests/apps
+
+            # update $PYTHONPATH so 'import eessi.testsuite.utils' works
+            export PYTHONPATH=$PWD:$PYTHONPATH
+            echo $PYTHONPATH
+            python -c 'import eessi.testsuite.utils'
+
+            # show active ReFrame configuration,
+            # enable verbose output to help expose problems with configuration file (if any)
+            reframe -vvv --show-config
+
+            # perform a dry run of *all* tests, without any filtering
+            time reframe --dry-run 2>&1 | tee dry_run.out
+            pattern='PASSED .*0 failure'
+            grep "${pattern}" dry_run.out || (echo "Pattern '${pattern}' not found!" && exit 1)
+
+            # list tests that are tagged with both 'CI' and '1_node'
+            time reframe --tag CI --tag 1_node --list-detailed 2>&1 | tee ci_1-node_tests_list.txt
+            # GROMACS CI test is HECBioSim/Crambin input
+            pattern='benchmark_info=HECBioSim/Crambin'
+            grep "${pattern}" ci_1-node_tests_list.txt || (echo "Pattern '${pattern}' not found!" && exit 1)

README.md

@@ -17,49 +17,52 @@ Alternatively, you can clone the repository
 git clone [email protected]:EESSI/test-suite.git
 ```
 
-and set your `PYTHONPATH` so that it includes the `eessi/reframe` directory from the repository.
+- add the path of the `test-suite` directory to your ``$PYTHONPATH``
 
 - create a site configuration file
 
-    - should look similar to `test-suite/eessi/reframe/config/settings_example.py`
+    - should look similar to `test-suite/config/settings_example.py`
 
 - run the tests
 
-    the example below runs a gromacs simulation using GROMACS modules available in the system,
-    in combination with all available system:partitions as defined in the site config file
+    The example below runs a gromacs simulation using GROMACS modules available
+    in the system, in combination with all available system:partitions as
+    defined in the site config file, using 1 full node (`--tag 1_node`, see `SCALES`
+    in `constants.py`).  This example assumes that you have cloned the
+    repository at `/path/to/EESSI/test-suite`.
 
 ```
-module load ReFrame/4.0.1
+cd /path/to/EESSI/test-suite
 
-eessiroot=<path_to_test-suite>
-eessihome=$eessiroot/eessi/reframe
+module load ReFrame/4.2.0
 
-PYTHONPATH=$PYTHONPATH:$EBROOTREFRAME:$eessihome reframe \
-    -C <path_to_site_config_file> \
-    -c $eessihome/eessi_checks/applications/ \
-    -t CI -t 1_node \
-    -r --performance-report
+export PYTHONPATH=$PWD:$PYTHONPATH
+
+reframe \
+    --config-file <path_to_site_config_file> \
+    --checkpath eessi/testsuite/tests/apps \
+    --tag CI --tag 1_node \
+    --run --performance-report
 ```
 
 ## Configuring GPU/non-GPU partitions in your site config file:
 
 - running GPU jobs in GPU nodes
-    - add feature `gpu` to the GPU partitions
+    - add `'features': [FEATURES[GPU]]` to the GPU partitions
+    - add `'extras': {GPU_VENDOR: GPU_VENDORS[NVIDIA]}` to the GPU partitions (or
+      `INTEL` or `AMD`, see `GPU_VENDORS` in `constants.py`)
 
 - running non-GPU jobs in non-GPU nodes
-    - add feature `cpu` to the non-GPU partitions
+    - add `'features': [FEATURES[CPU]]` to the non-GPU partitions
 
-- running GPU jobs and non-GPU jobs on gpu nodes
-    - add both features `cpu` and `gpu` to the GPU partitions
-    ```
-    'features': ['cpu', 'gpu'],
-    ```
+- running both GPU jobs and non-GPU jobs in GPU nodes
+    - add `'features': [FEATURES[CPU], FEATURES[GPU]]` to the GPU partitions
 
 - setting the number of GPUS per node <x> for a partition:
     ```
     'access': ['-p <partition_name>'],
     'devices': [
-        {'type': 'gpu', 'num_devices': <x>}
+        {'type': DEVICE_TYPES[GPU], 'num_devices': <x>}
     ],
     ```
 - requesting GPUs per node for a partition:
@@ -77,9 +80,13 @@ PYTHONPATH=$PYTHONPATH:$EBROOTREFRAME:$eessihome reframe \
 - specifying modules
     - `--setvar modules=<modulename>`
 
-- specifying systems:partitions
+- specifying valid systems:partitions
     - `--setvar valid_systems=<comma-separated-list>`
 
+      Note that setting `valid_systems` on the cmd line disables filtering of
+      valid systems:partitions in the hooks, so you have to do the filtering
+      yourself.
+
 - overriding tasks, cpus, gpus
     - `--setvar num_tasks_per_node=<x>`
     - `--setvar num_cpus_per_task=<y>`
@@ -88,17 +95,92 @@ PYTHONPATH=$PYTHONPATH:$EBROOTREFRAME:$eessihome reframe \
 - setting additional environment variables
     - `--setvar env_vars=<envar>:<value>`
 
+Note that these override the variables for _all_ tests in the test suite that
+respect those variables. To override a variable only for specific tests, one
+can use the `TEST.VAR` syntax. For example, to run the `GROMACS_EESSI` test with the
+module `GROMACS/2021.6-foss-2022a`:
+
+- `--setvar GROMACS_EESSI.modules=GROMACS/2021.6-foss-2022a`
+
 ## Developers
-If you want to install the EESSI test suite from a branch, you can use
+If you want to install the EESSI test suite from a branch, you can either
+install the feature branch with `pip`, or clone the Github repository and check
+out the feature branch.
+
+### Install from branch with pip
+
+To install from one of the branches of the main repository, use:
 
 ```bash
 pip install git+https://github.com/EESSI/test-suite.git@branchname
 ```
 
-This also works on forked repositories, e.g.
+Generally, you'll want to do this from a forked repository though, where
+someone worked on a feature. E.g.
 
 ```bash
 pip install git+https://github.com/<someuser>/test-suite.git@branchname
 ```
 
-making it an easy way of testing PRs.
+### Check out a feature branch from a fork
+We'll assume you already have a local clone of the official test-suite
+repository, called 'origin'. In that case, executing `git remote -v`, you
+should see:
+
+```bash
+$ git remote -v
+origin  [email protected]:EESSI/test-suite.git (fetch)
+origin  [email protected]:EESSI/test-suite.git (push)
+```
+
+You can add a fork to your local clone by adding a new remote. Pick a name for
+the remote that you find easy to recognize. E.g. to add the fork
+https://github.com/casparvl/test-suite and give it the (local) name `casparvl`,
+run:
+
+```bash
+git remote add casparvl [email protected]:casparvl/test-suite.git
+```
+
+With `git remote -v` you should now see the new remote:
+
+```bash
+$ git remote -v
+origin  [email protected]:EESSI/test-suite.git (fetch)
+origin  [email protected]:EESSI/test-suite.git (push)
+casparvl        [email protected]:casparvl/test-suite.git (fetch)
+casparvl        [email protected]:casparvl/test-suite.git (push)
+```
+
+Next, we'll fetch the branches that `casparvl` has in his fork:
+
+```bash
+$ git fetch casparvl
+```
+
+We can check the remote branches using
+```bash
+$ git branch --list --remotes
+  casparvl/gromacs_cscs
+  casparvl/main
+  casparvl/setuppy
+  casparvl/updated_defaults_pr11
+  origin/HEAD -> origin/main
+  origin/main
+```
+
+(remember to re-run `git fetch <remote>` if new branches don't show up with
+this command).
+
+Finally, we can create a new local branch (`-c`) and checkout one of these
+feature branches (e.g. `setuppy` from the remote `casparvl`). Here, we've
+picked `local_setuppy_branch` as the local branch name:
+```bash
+$ git switch -c local_setuppy_branch casparvl/setuppy
+```
+
+While the initial setup is a bit more involved, the advantage of this approach
+is that it is easy to pull in updates from a feature branch using `git pull`.
+You can also push back changes to the feature branch directly, but note that
+you are pushing to the Github fork of another Github user, so _make sure they
+are ok with that_ before doing so!