CSD3: Modify ReFrame configuration

Always request all of the memory on a node as `--exclusive` does not
imply this according to the current SLURM documentation
(https://slurm.schedmd.com/sbatch.html#OPT_exclusive). This can make a
difference to benchmark results when not using all of the cores on a
node due to the increased memory bandwidth available.

Also increase the job_submit_timeout as the SLURM controller can be a
bit slow on CSD3.

benchmarks/reframe_config.py

@@ -83,7 +83,10 @@ def spack_root_to_path():
                         ['I_MPI_OFI_PROVIDER', 'mlx'],
                         ['UCX_NET_DEVICES', 'mlx5_0:1'],
                     ],
-                    'access': ['--partition=icelake', '--exclusive'],
+                    'access': ['--partition=icelake', '--exclusive', '--mem=256120'],
+                    'sched_options': {
+                        'job_submit_timeout': 120,
+                    },
                     'environs': ['default'],
                     'max_jobs': 64,
                     'processor': {
@@ -94,7 +97,6 @@ def spack_root_to_path():
                     },
                 },
                 {
-                    # https://docs.hpc.cam.ac.uk/hpc/user-guide/icelake.html
                     'name': 'sapphirerapids',
                     'descr': 'Sapphire Rapids compute nodes',
                     'scheduler': 'slurm',
@@ -104,7 +106,10 @@ def spack_root_to_path():
                         ['I_MPI_OFI_PROVIDER', 'mlx'],
                         ['UCX_NET_DEVICES', 'mlx5_0:1'],
                     ],
-                    'access': ['--partition=sapphire', '--exclusive'],
+                    'access': ['--partition=sapphire', '--exclusive', '--mem=512960'],
+                    'sched_options': {
+                        'job_submit_timeout': 120,
+                    },
                     'environs': ['default'],
                     'max_jobs': 64,
                     'processor': {
@@ -134,7 +139,10 @@ def spack_root_to_path():
                         ['I_MPI_OFI_PROVIDER', 'mlx'],
                         ['UCX_NET_DEVICES', 'mlx5_0:1'],
                     ],
-                    'access': ['--partition=cclake', '--exclusive'],
+                    'access': ['--partition=cclake', '--exclusive', '--mem=191520'],
+                    'sched_options': {
+                        'job_submit_timeout': 120,
+                    },
                     'environs': ['default'],
                     'max_jobs': 64,
                     'processor': {