Replace quartz with dane (ECP-WarpX#5507)

The Quartz system at LLNL was removed and replaced by Dane. This PR
updates the scripts and install instructions for the new machine.

Docs/source/install/hpc.rst

@@ -50,7 +50,7 @@ This section documents quick-start guides for a selection of supercomputers that
    hpc/perlmutter
    hpc/pitzer
    hpc/polaris
-   hpc/quartz
+   hpc/dane
    hpc/summit
    hpc/taurus
    hpc/tioga

Docs/source/install/hpc/quartz.rst → Docs/source/install/hpc/dane.rst

@@ -1,27 +1,25 @@
-.. _building-quartz:
+.. _building-dane:
 
-Quartz (LLNL)
+Dane (LLNL)
 =============
 
-The `Quartz Intel CPU cluster <https://hpc.llnl.gov/hardware/platforms/quartz>`_ is located at LLNL.
+The `Dane Intel CPU cluster <https://hpc.llnl.gov/hardware/compute-platforms/dane>`_ is located at LLNL.
 
 
 Introduction
 ------------
 
 If you are new to this system, **please see the following resources**:
 
-* `LLNL user account <https://lc.llnl.gov/lorenz/mylc/mylc.cgi>`__ (login required)
-* `Quartz user guide <https://computing.llnl.gov/tutorials/linux_clusters/>`_
-* Batch system: `Slurm <https://computing.llnl.gov/tutorials/moab/>`_
+* `LLNL user account <https://lc.llnl.gov`__ (login required)
 * `Jupyter service <https://lc.llnl.gov/jupyter>`__ (`documentation <https://lc.llnl.gov/confluence/display/LC/JupyterHub+and+Jupyter+Notebook>`__, login required)
 * `Production directories <https://hpc.llnl.gov/hardware/file-systems>`_:
 
   * ``/p/lustre1/$(whoami)`` and ``/p/lustre2/$(whoami)``: personal directory on the parallel filesystem
   * Note that the ``$HOME`` directory and the ``/usr/workspace/$(whoami)`` space are NFS mounted and *not* suitable for production quality data generation.
 
 
-.. _building-quartz-preparation:
+.. _building-dane-preparation:
 
 Preparation
 -----------
@@ -32,23 +30,23 @@ Use the following commands to download the WarpX source code:
 
    git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
 
-We use system software modules, add environment hints and further dependencies via the file ``$HOME/quartz_warpx.profile``.
+We use system software modules, add environment hints and further dependencies via the file ``$HOME/dane_warpx.profile``.
 Create it now:
 
 .. code-block:: bash
 
-   cp $HOME/src/warpx/Tools/machines/quartz-llnl/quartz_warpx.profile.example $HOME/quartz_warpx.profile
+   cp $HOME/src/warpx/Tools/machines/dane-llnl/dane_warpx.profile.example $HOME/dane_warpx.profile
 
 .. dropdown:: Script Details
    :color: light
    :icon: info
    :animate: fade-in-slide-down
 
-   .. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz_warpx.profile.example
+   .. literalinclude:: ../../../../Tools/machines/dane-llnl/dane_warpx.profile.example
       :language: bash
 
 Edit the 2nd line of this script, which sets the ``export proj=""`` variable.
-For example, if you are member of the project ``tps``, then run ``vi $HOME/quartz_warpx.profile``.
+For example, if you are member of the project ``tps``, then run ``vi $HOME/dane_warpx.profile``.
 Enter the edit mode by typing ``i`` and edit line 2 to read:
 
 .. code-block:: bash
@@ -59,29 +57,29 @@ Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit).
 
 .. important::
 
-   Now, and as the first step on future logins to Quartz, activate these environment settings:
+   Now, and as the first step on future logins to Dane, activate these environment settings:
 
    .. code-block:: bash
 
-      source $HOME/quartz_warpx.profile
+      source $HOME/dane_warpx.profile
 
-Finally, since Quartz does not yet provide software modules for some of our dependencies, install them once:
+Finally, since Dane does not yet provide software modules for some of our dependencies, install them once:
 
 .. code-block:: bash
 
-   bash $HOME/src/warpx/Tools/machines/quartz-llnl/install_dependencies.sh
-   source /usr/workspace/${USER}/quartz/venvs/warpx-quartz/bin/activate
+   bash $HOME/src/warpx/Tools/machines/dane-llnl/install_dependencies.sh
+   source /usr/workspace/${USER}/dane/venvs/warpx-dane/bin/activate
 
 .. dropdown:: Script Details
    :color: light
    :icon: info
    :animate: fade-in-slide-down
 
-   .. literalinclude:: ../../../../Tools/machines/quartz-llnl/install_dependencies.sh
+   .. literalinclude:: ../../../../Tools/machines/dane-llnl/install_dependencies.sh
       :language: bash
 
 
-.. _building-quartz-compilation:
+.. _building-dane-compilation:
 
 Compilation
 -----------
@@ -91,27 +89,27 @@ Use the following :ref:`cmake commands <building-cmake>` to compile the applicat
 .. code-block:: bash
 
    cd $HOME/src/warpx
-   rm -rf build_quartz
+   rm -rf build_dane
 
-   cmake -S . -B build_quartz -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
-   cmake --build build_quartz -j 6
+   cmake -S . -B build_dane -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
+   cmake --build build_dane -j 6
 
-The WarpX application executables are now in ``$HOME/src/warpx/build_quartz/bin/``.
+The WarpX application executables are now in ``$HOME/src/warpx/build_dane/bin/``.
 Additionally, the following commands will install WarpX as a Python module:
 
 .. code-block:: bash
 
-   rm -rf build_quartz_py
+   rm -rf build_dane_py
 
-   cmake -S . -B build_quartz_py -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
-   cmake --build build_quartz_py -j 6 --target pip_install
+   cmake -S . -B build_dane_py -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
+   cmake --build build_dane_py -j 6 --target pip_install
 
-Now, you can :ref:`submit Quartz compute jobs <running-cpp-quartz>` for WarpX :ref:`Python (PICMI) scripts <usage-picmi>` (:ref:`example scripts <usage-examples>`).
-Or, you can use the WarpX executables to submit Quartz jobs (:ref:`example inputs <usage-examples>`).
-For executables, you can reference their location in your :ref:`job script <running-cpp-quartz>` or copy them to a location in ``$PROJWORK/$proj/``.
+Now, you can :ref:`submit Dane compute jobs <running-cpp-dane>` for WarpX :ref:`Python (PICMI) scripts <usage-picmi>` (:ref:`example scripts <usage-examples>`).
+Or, you can use the WarpX executables to submit Dane jobs (:ref:`example inputs <usage-examples>`).
+For executables, you can reference their location in your :ref:`job script <running-cpp-dane>` or copy them to a location in ``$PROJWORK/$proj/``.
 
 
-.. _building-quartz-update:
+.. _building-dane-update:
 
 Update WarpX & Dependencies
 ---------------------------
@@ -135,34 +133,34 @@ If you already installed WarpX in the past and want to update it, start by getti
 
 And, if needed,
 
-- :ref:`update the quartz_warpx.profile file <building-quartz-preparation>`,
+- :ref:`update the dane_warpx.profile file <building-dane-preparation>`,
 - log out and into the system, activate the now updated environment profile as usual,
-- :ref:`execute the dependency install scripts <building-quartz-preparation>`.
+- :ref:`execute the dependency install scripts <building-dane-preparation>`.
 
-As a last step, clean the build directory ``rm -rf $HOME/src/warpx/build_quartz`` and rebuild WarpX.
+As a last step, clean the build directory ``rm -rf $HOME/src/warpx/build_dane`` and rebuild WarpX.
 
 
-.. _running-cpp-quartz:
+.. _running-cpp-dane:
 
 Running
 -------
 
-.. _running-cpp-quartz-CPUs:
+.. _running-cpp-dane-CPUs:
 
-Intel Xeon E5-2695 v4 CPUs
+Intel Sapphire Rapids CPUs
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Quartz at LLNL.
+The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Dane at LLNL.
 Replace descriptions between chevrons ``<>`` by relevant values, for instance ``<input file>`` could be ``plasma_mirror_inputs``.
 
-.. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz.sbatch
+.. literalinclude:: ../../../../Tools/machines/dane-llnl/dane.sbatch
    :language: bash
-   :caption: You can copy this file from ``Tools/machines/quartz-llnl/quartz.sbatch``.
+   :caption: You can copy this file from ``Tools/machines/dane-llnl/dane.sbatch``.
 
-To run a simulation, copy the lines above to a file ``quartz.sbatch`` and run
+To run a simulation, copy the lines above to a file ``dane.sbatch`` and run
 
 .. code-block:: bash
 
-   sbatch quartz.sbatch
+   sbatch dane.sbatch
 
 to submit the job.

Tools/machines/quartz-llnl/quartz.sbatch → Tools/machines/dane-llnl/dane.sbatch

@@ -15,15 +15,14 @@
 # one MPI rank per half-socket (see below)
 #SBATCH --tasks-per-node=2
 # request all logical (virtual) cores per half-socket
-#SBATCH --cpus-per-task=18
+#SBATCH --cpus-per-task=112
 
 
-# each Quartz node has 1 socket of Intel Xeon E5-2695 v4
-# each Xeon CPU is divided into 2 bus rings that each have direct L3 access
+# each Dane node has 2 sockets of Intel Sapphire Rapids with 56 cores each
 export WARPX_NMPI_PER_NODE=2
 
-# each MPI rank per half-socket has 9 physical cores
-#   or 18 logical (virtual) cores
+# each MPI rank per half-socket has 56 physical cores
+#   or 112 logical (virtual) cores
 # over-subscribing each physical core with 2x
 #   hyperthreading led to a slight (3.5%) speedup on Cori's Intel Xeon E5-2698 v3,
 #   so we do the same here
@@ -33,7 +32,7 @@ export WARPX_NMPI_PER_NODE=2
 #   for N>9, also equally over close-by logical cores
 export OMP_PROC_BIND=spread
 export OMP_PLACES=threads
-export OMP_NUM_THREADS=18
+export OMP_NUM_THREADS=112
 
 EXE="<path/to/executable>"  # e.g. ./warpx
 

Tools/machines/quartz-llnl/quartz_warpx.profile.example → Tools/machines/dane-llnl/dane_warpx.profile.example

@@ -2,7 +2,7 @@
 #export proj="<yourProjectNameHere>"  # edit this and comment in
 
 # required dependencies
-module load cmake/3.23.1
+module load cmake/3.26.3
 module load clang/14.0.6-magic
 module load mvapich2/2.3.7
 
@@ -15,38 +15,38 @@ module load boost/1.80.0
 # optional: for openPMD support
 module load hdf5-parallel/1.14.0
 
-SW_DIR="/usr/workspace/${USER}/quartz"
-export CMAKE_PREFIX_PATH=${SW_DIR}/c-blosc-1.21.1:$CMAKE_PREFIX_PATH
+SW_DIR="/usr/workspace/${USER}/dane"
+export CMAKE_PREFIX_PATH=${SW_DIR}/c-blosc-1.21.6:$CMAKE_PREFIX_PATH
 export CMAKE_PREFIX_PATH=${SW_DIR}/adios2-2.8.3:$CMAKE_PREFIX_PATH
 export PATH=${SW_DIR}/adios2-2.8.3/bin:${PATH}
 
 # optional: for PSATD in RZ geometry support
-export CMAKE_PREFIX_PATH=${SW_DIR}/blaspp-2024.05.31:$CMAKE_PREFIX_PATH
-export CMAKE_PREFIX_PATH=${SW_DIR}/lapackpp-2024.05.31:$CMAKE_PREFIX_PATH
-export LD_LIBRARY_PATH=${SW_DIR}/blaspp-2024.05.31/lib64:$LD_LIBRARY_PATH
-export LD_LIBRARY_PATH=${SW_DIR}/lapackpp-2024.05.31/lib64:$LD_LIBRARY_PATH
+export CMAKE_PREFIX_PATH=${SW_DIR}/blaspp-2024.10.26:$CMAKE_PREFIX_PATH
+export CMAKE_PREFIX_PATH=${SW_DIR}/lapackpp-2024.10.26:$CMAKE_PREFIX_PATH
+export LD_LIBRARY_PATH=${SW_DIR}/blaspp-2024.10.26/lib64:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH=${SW_DIR}/lapackpp-2024.10.26/lib64:$LD_LIBRARY_PATH
 
 # optional: for Python bindings
-module load python/3.9.12
+module load python/3.12.2
 
-if [ -d "${SW_DIR}/venvs/warpx-quartz" ]
+if [ -d "${SW_DIR}/venvs/warpx-dane" ]
 then
-    source ${SW_DIR}/venvs/warpx-quartz/bin/activate
+    source ${SW_DIR}/venvs/warpx-dane/bin/activate
 fi
 
 # optional: an alias to request an interactive node for two hours
-alias getNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=18 -p pdebug --pty bash"
+alias getNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=56 -p pdebug --pty bash"
 # an alias to run a command on a batch node for up to 30min
 #   usage: runNode <command>
-alias runNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=18 -p pdebug"
+alias runNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=56 -p pdebug"
 
 # fix system defaults: do not escape $ with a \ on tab completion
 shopt -s direxpand
 
-# optimize CPU microarchitecture for Intel Xeon E5-2695 v4
+# optimize CPU microarchitecture for Intel Sapphire Rapids
 # note: the cc/CC/ftn wrappers below add those
-export CXXFLAGS="-march=broadwell"
-export CFLAGS="-march=broadwell"
+export CXXFLAGS="-march=sapphirerapids"
+export CFLAGS="-march=sapphirerapids"
 
 # compiler environment hints
 export CC=$(which clang)

Tools/machines/quartz-llnl/install_dependencies.sh → Tools/machines/dane-llnl/install_dependencies.sh

@@ -1,10 +1,10 @@
 #!/bin/bash
 #
-# Copyright 2023 The WarpX Community
+# Copyright 2024 The WarpX Community
 #
 # This file is part of WarpX.
 #
-# Author: Axel Huebl
+# Author: Axel Huebl, David Grote
 # License: BSD-3-Clause-LBNL
 
 # Exit on first error encountered #############################################
@@ -14,13 +14,13 @@ set -eu -o pipefail
 
 # Check: ######################################################################
 #
-#   Was quartz_warpx.profile sourced and configured correctly?
-if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your quartz_warpx.profile file! Please edit its line 2 to continue!"; exit 1; fi
+#   Was dane_warpx.profile sourced and configured correctly?
+if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your dane_warpx.profile file! Please edit its line 2 to continue!"; exit 1; fi
 
 
 # Remove old dependencies #####################################################
 #
-SW_DIR="/usr/workspace/${USER}/quartz"
+SW_DIR="/usr/workspace/${USER}/dane"
 rm -rf ${SW_DIR}
 mkdir -p ${SW_DIR}
 
@@ -41,13 +41,13 @@ if [ -d ${HOME}/src/c-blosc ]
 then
   cd ${HOME}/src/c-blosc
   git fetch --prune
-  git checkout v1.21.1
+  git checkout v1.21.6
   cd -
 else
-  git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git ${HOME}/src/c-blosc
+  git clone -b v1.21.6 https://github.com/Blosc/c-blosc.git ${HOME}/src/c-blosc
 fi
-cmake -S ${HOME}/src/c-blosc -B ${build_dir}/c-blosc-quartz-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1
-cmake --build ${build_dir}/c-blosc-quartz-build --target install --parallel 6
+cmake -S ${HOME}/src/c-blosc -B ${build_dir}/c-blosc-dane-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.6
+cmake --build ${build_dir}/c-blosc-dane-build --target install --parallel 6
 
 # ADIOS2
 if [ -d ${HOME}/src/adios2 ]
@@ -59,44 +59,44 @@ then
 else
   git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git ${HOME}/src/adios2
 fi
-cmake -S ${HOME}/src/adios2 -B ${build_dir}/adios2-quartz-build -DBUILD_TESTING=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3
-cmake --build ${build_dir}/adios2-quartz-build --target install -j 6
+cmake -S ${HOME}/src/adios2 -B ${build_dir}/adios2-dane-build -DBUILD_TESTING=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3
+cmake --build ${build_dir}/adios2-dane-build --target install -j 6
 
 # BLAS++ (for PSATD+RZ)
 if [ -d ${HOME}/src/blaspp ]
 then
   cd ${HOME}/src/blaspp
   git fetch --prune
-  git checkout v2024.05.31
+  git checkout v2024.10.26
   cd -
 else
-  git clone -b v2024.05.31 https://github.com/icl-utk-edu/blaspp.git ${HOME}/src/blaspp
+  git clone -b v2024.10.26 https://github.com/icl-utk-edu/blaspp.git ${HOME}/src/blaspp
 fi
-cmake -S ${HOME}/src/blaspp -B ${build_dir}/blaspp-quartz-build -Duse_openmp=ON -Duse_cmake_find_blas=ON -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-2024.05.31
-cmake --build ${build_dir}/blaspp-quartz-build --target install --parallel 6
+cmake -S ${HOME}/src/blaspp -B ${build_dir}/blaspp-dane-build -Duse_openmp=ON -Duse_cmake_find_blas=ON -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-2024.10.26
+cmake --build ${build_dir}/blaspp-dane-build --target install --parallel 6
 
 # LAPACK++ (for PSATD+RZ)
 if [ -d ${HOME}/src/lapackpp ]
 then
   cd ${HOME}/src/lapackpp
   git fetch --prune
-  git checkout v2024.05.31
+  git checkout v2024.10.26
   cd -
 else
-  git clone -b v2024.05.31 https://github.com/icl-utk-edu/lapackpp.git ${HOME}/src/lapackpp
+  git clone -b v2024.10.26 https://github.com/icl-utk-edu/lapackpp.git ${HOME}/src/lapackpp
 fi
-CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S ${HOME}/src/lapackpp -B ${build_dir}/lapackpp-quartz-build -Duse_cmake_find_lapack=ON -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-2024.05.31
-cmake --build ${build_dir}/lapackpp-quartz-build --target install --parallel 6
+CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S ${HOME}/src/lapackpp -B ${build_dir}/lapackpp-dane-build -Duse_cmake_find_lapack=ON -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-2024.10.26
+cmake --build ${build_dir}/lapackpp-dane-build --target install --parallel 6
 
 
 # Python ######################################################################
 #
 python3 -m pip install --upgrade --user virtualenv
-rm -rf ${SW_DIR}/venvs/warpx-quartz
-python3 -m venv ${SW_DIR}/venvs/warpx-quartz
-source ${SW_DIR}/venvs/warpx-quartz/bin/activate
+rm -rf ${SW_DIR}/venvs/warpx-dane
+python3 -m venv ${SW_DIR}/venvs/warpx-dane
+source ${SW_DIR}/venvs/warpx-dane/bin/activate
 python3 -m pip install --upgrade pip
-python3 -m pip cache purge
+#python3 -m pip cache purge
 python3 -m pip install --upgrade build
 python3 -m pip install --upgrade packaging
 python3 -m pip install --upgrade wheel