implement better support for cmd line specified valid_systems

[smoors]

currently, if you specify valid_systems on the command line, no automatic filtering is done, so you have to do the filtering yourself.

the reason is that unfortunately, combining systemname:partitionname with features is not fully supported in the current version of ReFrame (4.0.5).

for example, this works (multiple list items behave like the OR operation):

valid_partitions = ['*:ampere', '+cpu']

but this does not (multiple string items behave like AND operation):

valid_partitions = ['*:ampere +cpu']

this is what ReFrame shows in this case:

WARNING: skipping test 'GROMACS_EESSI': type error: test-suite/eessi/reframe/eessi-checks/applications/gromacs_check.py:62: failed to set field 'valid_systems': '['*:ampere +gpu']' is not of type 'List[Str[r'^(((\*|(\w[-.\w]*))(:(\*|(\w[-.\w]*)))?)|(([+-](\w[-.\w]*))|(%(\w[-.\w]*)=\S+))(\s+(([+-](\w[-.\w]*))|(%(\w[-.\w]*)=\S+)))*)$']]'
                self.valid_systems = [valid_systems]

maybe it's possible check the features of the partitions that are specified on the cmd line, and filter the tests based on that

more info here: https://reframe-hpc.readthedocs.io/en/stable/regression_test_api.html#reframe.core.pipeline.RegressionTest.valid_systems

[smoors]

i'll create an issue for this in ReFrame as well, maybe they can solve this.

[satishskamath]

[EESSI pilot 2021.06] $ reframe -C /home/satishk/projects/eessi_reframe/settings_example.py -c $eessihome/eessi-checks/applications/ -t CI -t singlen
ode -l --performance-report --setvar valid_systems=snellius_eessi:cpu
[ReFrame Setup]
  version:           4.0.1
  command:           '/sw/arch/RHEL8/EB_production/2022/software/ReFrame/4.0.1/bin/reframe -C /home/satishk/projects/eessi_reframe/settings_example.p
y -c /gpfs/home5/satishk/projects/test-suite/eessi/reframe/eessi-checks/applications/ -t CI -t singlenode -l --performance-report --setvar valid_syst
ems=snellius_eessi:cpu'
  launched by:       [email protected]
  working directory: '/gpfs/home5/satishk/projects/eessi_reframe'
  settings files:    '<builtin>', '/home/satishk/projects/eessi_reframe/settings_example.py'
  check search path: '/gpfs/home5/satishk/projects/test-suite/eessi/reframe/eessi-checks/applications'
  stage directory:   '/scratch-shared/satishk/reframe_output/staging'
  output directory:  '/gpfs/home5/satishk/projects/eessi_reframe/output'
  log files:         '/gpfs/home5/satishk/projects/eessi_reframe/reframe.log'

[List of matched checks]
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=gpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a /ff91f18a
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=gpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a-CUDA-11.7.0 /54be99
0c
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=gpu %scale=('singlenode', 1) %module_name=GROMACS/2020.4-foss-2020a-Python-3.8.2 /0972b
423
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=gpu %scale=('singlenode', 1) %module_name=GROMACS/2020.1-foss-2020a-Python-3.8.2 /b2862
df7
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a /d12ed623
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2021.6-foss-2022a-CUDA-11.7.0 /a6f0d1
0c
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2020.4-foss-2020a-Python-3.8.2 /80c7f
af7
- GROMACS_EESSI %benchmark_info=HECBioSim/hEGFRDimer %nb_impl=cpu %scale=('singlenode', 1) %module_name=GROMACS/2020.1-foss-2020a-Python-3.8.2 /a4450
ec7
Found 8 check(s)

Log file(s) saved in '/gpfs/home5/satishk/projects/eessi_reframe/reframe.log'
[EESSI pilot 2021.06] $ 

Transferring the example that I checked, here.

[smoors]

upstream issue: reframe-hpc/reframe#2820