diff --git a/docs/software-tools/ddt.md b/docs/software-tools/ddt.md index 0838dbd4..5685b195 100644 --- a/docs/software-tools/ddt.md +++ b/docs/software-tools/ddt.md @@ -1,13 +1,13 @@ -# Debugging using Arm DDT +# Debugging using Linaro DDT -The Arm Forge tool suite is installed on Cirrus. This includes DDT, +The Linaro Forge tool suite is installed on Cirrus. This includes DDT, which is a debugging tool for scalar, multi-threaded and large-scale parallel applications. To compile your code for debugging you will usually want to specify the `-O0` option to turn off all code optimisation (as this can produce a mismatch between source code line numbers and debugging information) and `-g` to include debugging information in the compiled executable. To use this package you will -need to log in to Cirrus with X11-forwarding enabled, load the Arm Forge +need to log in to Cirrus with X11-forwarding enabled, load the Linaro Forge module and execute `forge`: module load forge @@ -37,7 +37,7 @@ tick the *MPI* box -- when running on the compute nodes, you must set the MPI implementation to *Slurm (generic)*. You must also tick the *Submit to Queue* box. Clicking the *Configure* button in this section, you must now choose the submission template. One is provided for you at -`/mnt/lustre/indy2lfs/sw/arm/forge/latest/templates/cirrus.qtf` which +`/work/y07/shared/cirrus-software/forge/latest/templates/cirrus.qtf` which you should copy and modify to suit your needs. You will need to load any modules required for your code and perform any other necessary setup, such as providing extra sbatch options, i.e., whatever is needed for @@ -47,7 +47,7 @@ your code to run in a normal batch job. !!! Note - The current Arm Forge licence permits use on the Cirrus CPU nodes only. + The current Linaro Forge licence permits use on the Cirrus CPU nodes only. The licence does not permit use of DDT/MAP for codes that run on the Cirrus GPUs. @@ -77,18 +77,18 @@ libraries should be found without further arguments. ## Remote Client -Arm Forge can connect to remote systems using SSH so you can run the +Linaro Forge can connect to remote systems using SSH so you can run the user interface on your desktop or laptop machine without the need for X forwarding. Native remote clients are available for Windows, macOS and -Linux. You can download the remote clients from the [Arm -website](https://developer.arm.com/downloads/-/arm-forge). No licence +Linux. You can download the remote clients from the [Linaro Forge +website](https://www.linaroforge.com/downloadForge/). No licence file is required by a remote client. !!! Note - The same versions of Arm Forge must be installed on the local and remote + The same versions of Linaro Forge must be installed on the local and remote systems in order to use DDT remotely. @@ -98,7 +98,7 @@ click on the *Remote Launch* drop-down box and click on *Configure*. In the new window, click *Add* to create a new login profile. For the hostname you should provide `username@login.cirrus.ac.uk` where `username` is your login username. For *Remote Installation Directory*\* -enter `/mnt/lustre/indy2lfs/sw/arm/forge/latest`. To ensure your SSH +enter `/work/y07/shared/cirrus-software/forge/latest`. To ensure your SSH private key can be used to connect, the SSH agent on your local machine should be configured to provide it. You can ensure this by running `ssh-add ~/.ssh/id_rsa_cirrus` before using the Forge client where you @@ -127,11 +127,11 @@ usual login password the connection to Cirrus will be established and you will be able to start debugging. You can find more detailed information -[here](https://developer.arm.com/documentation/101136/2011/Arm-Forge/Connecting-to-a-remote-system). +[here](https://docs.linaroforge.com/23.1.1/html/forge/forge/connecting_to_a_remote_system/connecting_remotely.html). ## Getting further help on DDT - [DDT - website](https://www.arm.com/products/development-tools/server-and-hpc/forge/ddt) + website](https://www.linaroforge.com/linaroDdt/) - [DDT user - guide](https://developer.arm.com/documentation/101136/22-1-3/DDT?lang=en) + guide](https://docs.linaroforge.com/23.1.1/html/forge/ddt/index.html) diff --git a/docs/user-guide/development.md b/docs/user-guide/development.md index 54c2bea6..0552119d 100644 --- a/docs/user-guide/development.md +++ b/docs/user-guide/development.md @@ -42,10 +42,10 @@ You can list all the modules of a particular type by providing an argument to the `module avail` command. For example, to list all available versions of the Intel Compiler type: - [user@cirrus-login0 ~]$ module avail intel-compilers + [user@cirrus-login0 ~]$ module avail intel-*/compilers - --------------------------------- /mnt/lustre/indy2lfs/sw/modulefiles -------------------------------- - intel-compilers-18/18.05.274 intel-compilers-19/19.0.0.117 + --------------------------------- /work/y07/shared/cirrus-modulefiles -------------------------------- + intel-19.5/compilers intel-20.4/compilers If you want more info on any of the modules, you can use the `module help` command: @@ -66,46 +66,37 @@ their versions you have presently loaded in your environment, e.g.: [user@cirrus-login0 ~]$ module list Currently Loaded Modulefiles: - 1) git/2.35.1(default) 6) gcc/8.2.0(default) - 2) singularity/3.7.2(default) 7) intel-cc-18/18.0.5.274 - 3) epcc/utils 8) intel-fc-18/18.0.5.274 - 4) /mnt/lustre/indy2lfs/sw/modulefiles/epcc/setup-env 9) intel-compilers-18/18.05.274 - 5) intel-license 10) mpt/2.25 + 1) git/2.35.1(default) + 2) epcc/utils + 2) /mnt/lustre/e1000/home/y07/shared/cirrus-modulefiles/epcc/setup-env ### Loading, unloading and swapping modules To load a module to use `module add` or `module load`. For example, to -load the intel-compilers-18 into the development environment: +load the intel 20.4 compilers into the development environment: - module load intel-compilers-18 + module load intel-20.4/compilers -This will load the default version of the intel compilers. If you need a -specific version of the module, you can add more information: - - module load intel-compilers-18/18.0.5.274 - -will load version 18.0.2.274 for you, regardless of the default. +This will load the default version of the intel compilers. If a module loading file cannot be accessed within 10 seconds, a warning message will appear: `Warning: Module system not loaded`. If you want to clean up, `module remove` will remove a loaded module: - module remove intel-compilers-18 + module remove intel-20.4/compilers -(or `module rm intel-compilers-18` or -`module unload intel-compilers-18`) will unload what ever version of -intel-compilers-18 (even if it is not the default) you might have -loaded. There are many situations in which you might want to change the +You could also run `module rm intel-20.4/compilers` or `module unload intel-20.4/compilers`. +There are many situations in which you might want to change the presently loaded version to a different one, such as trying the latest version which is not yet the default or using a legacy version to keep compatibility with old data. This can be achieved most easily by using "module swap oldmodule newmodule". -Suppose you have loaded version 18 of the Intel compilers; the following -command will change to version 19: +Suppose you have loaded version 19 of the Intel compilers; the following +command will change to version 20: - module swap intel-compilers-18 intel-compilers-19 + module swap intel-19.5/compilers intel-20.4/compilers ## Available Compiler Suites @@ -119,11 +110,11 @@ command will change to version 19: ### Intel Compiler Suite -The Intel compiler suite is accessed by loading the `intel-compilers-*` -and `intel-*/compilers` modules, where `*` references the version. For -example, to load the 2019 release, you would run: +The Intel compiler suite is accessed by loading the `intel-*/compilers` +module, where `*` references the version. For example, to load the v20 +release, you would run: - module load intel-compilers-19 + module load intel-20.4/compilers Once you have loaded the module, the compilers are available as: @@ -137,10 +128,10 @@ compiler versions and tools. ### GCC Compiler Suite The GCC compiler suite is accessed by loading the `gcc/*` modules, where -`*` again is the version. For example, to load version 8.2.0 you would +`*` again is the version. For example, to load version 10.2.0 you would run: - module load gcc/8.2.0 + module load gcc/10.2.0 Once you have loaded the module, the compilers are available as: @@ -193,9 +184,9 @@ use to compile your code. #### Using Intel Compilers and HPE MPT Once you have loaded the MPT module you should next load the Intel -compilers module you intend to use (e.g. `intel-compilers-19`): +compilers module you intend to use (e.g. `intel-20.4/compilers`): - module load intel-compilers-19 + module load intel-20.4/compilers The compiler wrappers are then available as @@ -243,9 +234,9 @@ Compilers are then available as Although HPE MPT remains the default MPI library and we recommend that first attempts at building code follow that route, you may also choose to use Intel MPI if you wish. To use these, load the appropriate -`intel-mpi` module, for example `intel-mpi-19`: +MPI module, for example `intel-20.4/mpi`: - module load intel-mpi-19 + module load intel-20.4/mpi Please note that the name of the wrappers to use when compiling with Intel MPI depends on whether you are using the Intel compilers or GCC. @@ -260,40 +251,12 @@ building software. - - -!!! Note - - - Using Intel MPI 18 can cause warnings in your output similar to - `no hfi units are available` or - `The /dev/hfi1_0 device failed to appear`. These warnings can be safely - ignored, or, if you would prefer to prevent them, you may add the line - - export I_MPI_FABRICS=shm:ofa - - to your job scripts after loading the Intel MPI 18 module. - - - - - -!!! Note - - - - When using Intel MPI 18, you should always launch MPI tasks with `srun`, - the supported method on Cirrus. Launches with `mpirun` or `mpiexec` will - likely fail. - - - #### Using Intel Compilers and Intel MPI After first loading Intel MPI, you should next load the appropriate -`intel-compilers` module (e.g. `intel-compilers-19`): +Intel compilers module (e.g. `intel-20.4/compilers`): - module load intel-compilers-19 + module load intel-20.4/compilers You may then use the following MPI compiler wrappers: @@ -328,7 +291,7 @@ specific CUDA version, one that is fully compatible with the underlying NVIDIA GPU device driver. See the link below for an example how an OpenMPI build is configured. -[Build instructions for OpenMPI 4.1.5 on Cirrus](https://github.com/hpc-uk/build-instructions/blob/main/libs/openmpi/build_openmpi_4.1.5_cirrus_gcc8.md) +[Build instructions for OpenMPI 4.1.6 on Cirrus](https://github.com/hpc-uk/build-instructions/blob/main/libs/openmpi/build_openmpi_4.1.6_cirrus_gcc10.md) All this means we build can OpenMPI such that it supports direct GPU-to-GPU communications using the NVLink intra-node GPU comm links (and inter-node GPU comms are direct to Infiniband @@ -442,8 +405,6 @@ A full list is available via `module avail intel`. The different available compiler versions are: -- `intel-*/18.0.5.274` Intel 2018 Update 4 -- `intel-*/19.0.0.117` Intel 2019 Initial release - `intel-19.5/*` Intel 2019 Update 5 - `intel-20.4/*` Intel 2020 Update 4 diff --git a/docs/user-guide/gpu.md b/docs/user-guide/gpu.md index d2413626..5b7b045b 100644 --- a/docs/user-guide/gpu.md +++ b/docs/user-guide/gpu.md @@ -44,10 +44,8 @@ are particular reasons to use earlier versions. The default version is therefore the latest module version present on the system. Each release of the NVIDIA HPC SDK may include several different -versions of the CUDA toolchain. For example, the `nvidia/nvhpc/21.2` -module comes with CUDA 10.2, 11.0 and 11.2. Only one of these CUDA -toolchains can be active at any one time and for `nvhpc/22.11` this is -CUDA 11.8. +versions of the CUDA toolchain. Only one of these CUDA toolchains +can be active at any one time and for `nvhpc/22.11` this is CUDA 11.8. Here is a list of available HPC SDK versions, and the corresponding version of CUDA: @@ -88,11 +86,11 @@ Compile your source code in the usual way. #### Using CUDA with Intel compilers -You can load either the Intel 18 or Intel 19 compilers to use with +You can load either the Intel 19 or Intel 20 compilers to use with `nvcc`. module unload gcc - module load intel-compilers-19 + module load intel-20.4/compilers You can now use `nvcc -ccbin icpc` to compile your source code with the Intel C++ compiler `icpc`. @@ -401,8 +399,8 @@ via the links below. -If your code was compiled with the tools provided by `nvidia/nvhpc/21.2` -you should download and install Nsight Systems v2020.5.1.85. +If your code was compiled with the tools provided by `nvidia/nvhpc/22.2` +you should download and install Nsight Systems v2023.4.1.97. ### Using Nsight Compute @@ -437,10 +435,10 @@ Consult the NVIDIA documentation for further details. - + -Nsight Compute v2021.3.1.0 has been found to work for codes compiled -using `nvhpc` versions 21.2 and 21.9. +Nsight Compute v2023.3.1.0 has been found to work for codes compiled +using `nvhpc` versions 22.2 and 22.11. ## Monitoring the GPU Power Usage @@ -526,8 +524,8 @@ bandwidth. Version of OpenMPI with both CUDA-aware MPI support and SLURM support are available, you should load the following modules: - module load openmpi/4.1.4-cuda-11.8 - module load nvidia/nvhpc-nompi/22.11 + module load openmpi/4.1.6-cuda-11.6 + module load nvidia/nvhpc-nompi/22.2 The command you use to compile depends on whether you are compiling C/C++ or Fortran. @@ -563,7 +561,7 @@ A batch script to use such an executable might be: #SBATCH --gres=gpu:4 # Load the appropriate modules, e.g., - module load openmpi/4.1.4-cuda-11.8 + module load openmpi/4.1.6-cuda-11.6 module load nvidia/nvhpc-nompi/22.2 export OMP_NUM_THREADS=1 diff --git a/docs/user-guide/python.md b/docs/user-guide/python.md index d7bfc286..e9d78679 100644 --- a/docs/user-guide/python.md +++ b/docs/user-guide/python.md @@ -5,16 +5,15 @@ Python on Cirrus is provided by a number of [Anaconda](https://www.continuum.io) module. (Miniconda being a small bootstrap version of Anaconda). -The Anaconda module is called `anaconda/python3` and is suitable for +The Anaconda module is called `anaconda3/2023.9` and is suitable for running serial applications - for parallel applications using `mpi4py` see [mpi4py for CPU](#mpi4py-for-cpu) or [mpi4py for GPU](#mpi4py-for-gpu). You can list the Miniconda modules by running `module avail python` on a login node. Those module versions that have the `gpu` suffix are suitable for use on the [Cirrus GPU nodes](../gpu). There are also -modules that extend these Python environments, e.g., `pyfr`, `horovod`, -`tensorflow` and `pytorch` - simply run `module help ` for -further info. +modules that extend these Python environments, e.g., `pyfr`, `tensorflow` +and `pytorch` - simply run `module help ` for further info. The Miniconda modules support Python-based parallel codes, i.e., each such `python` module provides a suite of packages pertinent to parallel @@ -27,8 +26,8 @@ can be used on the Cirrus CPU/GPU nodes. ## mpi4py for CPU -The `python/3.9.13` module provides mpi4py 3.1.3 linked with OpenMPI -4.1.4. +The `python/3.9.13` module provides mpi4py 3.1.5 linked with OpenMPI +4.1.6. See `numpy-broadcast.py` below which is a simple MPI Broadcast example, and the Slurm script `submit-broadcast.slurm` which demonstrates how to @@ -203,10 +202,10 @@ below for further details. There are several more Python-based modules that also target the Cirrus GPU nodes. These include two machine learning frameworks, -`pytorch/1.12.1-gpu` and `tensorflow/2.9.1-gpu`. Both modules are Python -virtual environments that extend `python/3.9.13-gpu`. The MPI comms is +`pytorch/1.13.1-gpu` and `tensorflow/2.15.0-gpu`. Both modules are Python +virtual environments that extend `python/3.10.8-gpu`. The MPI comms is handled by the [Horovod](https://horovod.readthedocs.io/en/stable/) -0.25.0 package along with the [NVIDIA Collective Communications +0.28.1 package along with the [NVIDIA Collective Communications Library](https://developer.nvidia.com/nccl) v2.11.4. A full package list for these environments can be obtained by loading @@ -214,7 +213,7 @@ the module of interest and then running `pip list`. Please click on the link indicated to see examples of how to use the [PyTorch and TensorFlow -modules](https://github.com/hpc-uk/build-instructions/blob/main/pyenvs/horovod/run_horovod_0.25.0_cirrus_gpu.md) +modules](https://github.com/hpc-uk/build-instructions/blob/main/pyenvs/ml/run_ml_workloads_cirrus_gpu.md) . ## Installing your own Python packages (with pip) @@ -324,10 +323,10 @@ srun --ntasks=8 --tasks-per-node=4 --cpus-per-task=10 myvenv-script.py Lastly, the environment being extended does not have to come from one of the centrally-installed `python` modules. You could just as easily create a local virtual environment based on one of the Machine Learning -(ML) modules, e.g., `horovod`, `tensorflow` or `pytorch`. This means you -would avoid having to install ML packages within your local area. Each -of those ML modules is based on a `python` module. For example, -`tensorflow/2.11.0-gpu` is itself an extension of `python/3.10.8-gpu`. +(ML) modules, e.g., `tensorflow` or `pytorch`. This means you would avoid +having to install ML packages within your local area. Each of those ML +modules is based on a `python` module. For example, `tensorflow/2.15.0-gpu` +is itself an extension of `python/3.10.8-gpu`. ## Installing your own Python packages (with conda) @@ -447,7 +446,7 @@ the single `source` command that is sufficient for a `pip` environment. Further, `conda` cannot be used if the base environment is one of the Machine Learning (ML) modules, as `conda` is not flexible enough to gather Python packages from both the ML and base `python` modules (e.g., -the ML module `pytorch/2.0.0-gpu` is itself based on +the ML module `pytorch/1.13.1-gpu` is itself based on `python/3.10.8-gpu`, and so `conda` will only duplicate packages provided by the `python` module and not the ones supplied by `pytorch`). diff --git a/docs/user-guide/singularity.md b/docs/user-guide/singularity.md index f8cc5faa..ee2d9c97 100644 --- a/docs/user-guide/singularity.md +++ b/docs/user-guide/singularity.md @@ -219,7 +219,7 @@ below shows that the `srun` command now contains an additional module load singularity # The host bind paths for the Singularity container. - BIND_ARGS=/mnt/lustre/indy2lfs/sw,/opt/hpe,/etc/libibverbs.d,/path/to/input/files + BIND_ARGS=/work/y07/shared/cirrus-software,/opt/hpe,/etc/libibverbs.d,/path/to/input/files # The file containing environment variable settings that will allow # the container to find libraries on the host, e.g., LD_LIBRARY_PATH . @@ -342,20 +342,20 @@ must be run on a system where you have root access, not on Cirrus). The resulting image file (`centos7.sif`) can then be copied to Cirrus using scp; such an image already exists on Cirrus and can be found in -the `/mnt/lustre/indy2lfs/sw/singularity/images` folder. +the `/work/y07/shared/cirrus-software/singularity/images` folder. When you use that image interactively on Cirrus you must start with a -login shell and also bind `/mnt/lustre/indy2lfs/sw` so that the container +login shell and also bind `/work/y07/shared/cirrus-software` so that the container can see all the module files, see below. [user@cirrus-login1 ~]$ module load singularity - [user@cirrus-login1 ~]$ singularity exec -B /mnt/lustre/indy2lfs/sw \ - /mnt/lustre/indy2lfs/sw/singularity/images/centos7.sif \ + [user@cirrus-login1 ~]$ singularity exec -B /work/y07/shared/cirrus-software \ + /work/y07/shared/cirrus-software/singularity/images/centos7.sif \ /bin/bash --login - Singularity> module avail intel-compilers + Singularity> module avail intel-*/compilers - --------- /mnt/lustre/indy2lfs/sw/modulefiles ------------- - intel-compilers-18/18.05.274 intel-compilers-19/19.0.0.117 + --------- /work/y07/shared/cirrus-modulefiles ------------- + intel-19.5/compilers intel-20.4/compilers Singularity> exit logout [user@cirrus-login1 ~]$ @@ -372,10 +372,10 @@ followed by a `shell` command with the `--writable` option. You are now free to change the files inside the container sandbox. user@cirrus-login1 ~]$ singularity build --sandbox image.sif.sandbox image.sif - user@cirrus-login1 ~]$ singularity shell -B /mnt/lustre/indy2lfs/sw --writable image.sif.sandbox + user@cirrus-login1 ~]$ singularity shell -B /work/y07/shared/cirrus-software --writable image.sif.sandbox Singularity> -In the example above, the `/mnt/lustre/indy2lfs/sw` bind path is specified, allowing +In the example above, the `/work/y07/shared/cirrus-software` bind path is specified, allowing you to build code that links to the Cirrus module libraries. Finally, once you are finished with the sandbox you can exit and convert diff --git a/mkdocs.yml b/mkdocs.yml index 2480eaf3..fad9a32f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -68,7 +68,7 @@ nav: - "Intel MKL: BLAS, LAPACK, ScaLAPACK": software-libraries/intel_mkl.md - "HDF5": software-libraries/hdf5.md - "Software Tools": - - "Debugging using Arm DDT": software-tools/ddt.md + - "Debugging using Linaro DDT": software-tools/ddt.md - "Profiling using Scalasca": software-tools/scalasca.md - "Intel VTune": software-tools/intel-vtune.md - "References and further reading": user-guide/reading.md