From fd65046fe117691f2de494933c1179a582815854 Mon Sep 17 00:00:00 2001 From: EdwardSnyder-NOAA <96196752+EdwardSnyder-NOAA@users.noreply.github.com> Date: Thu, 9 Jan 2025 16:17:07 -0600 Subject: [PATCH 1/8] Update Container.rst Add info about gaea rocoto requirements. --- doc/source/BuildingRunningTesting/Container.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/BuildingRunningTesting/Container.rst b/doc/source/BuildingRunningTesting/Container.rst index d895cca6..ec05676e 100644 --- a/doc/source/BuildingRunningTesting/Container.rst +++ b/doc/source/BuildingRunningTesting/Container.rst @@ -245,7 +245,7 @@ Configure the Experiment The user should now see the ``Land-DA_workflow`` and ``jedi-bundle`` directories in the ``$LANDDAROOT`` directory. -Because of a conda conflict between the container and the host system, it is best to load rocoto separately instead of using workflow files found in the ``modulefiles`` directory. +Because of a conda conflict between the container and the host system, it is best to load rocoto separately instead of using workflow files found in the ``modulefiles`` directory. Note that some modules need to be loaded or a path appended before running the following command. Check the ``modulefiles/wflow_.lua`` to see if that is the case. For Gaea, this path needs to be loaded first ``/ncrc/proj/epic/rocoto/modulefiles/``. .. code-block:: console From 1bc4bbc0e593f5254d205305ce49cf48da14e6c6 Mon Sep 17 00:00:00 2001 From: EdwardSnyder-NOAA <96196752+EdwardSnyder-NOAA@users.noreply.github.com> Date: Fri, 10 Jan 2025 09:23:04 -0600 Subject: [PATCH 2/8] Update Container.rst Add additional compilers and mpi options. --- doc/source/BuildingRunningTesting/Container.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/source/BuildingRunningTesting/Container.rst b/doc/source/BuildingRunningTesting/Container.rst index ec05676e..e5a3c312 100644 --- a/doc/source/BuildingRunningTesting/Container.rst +++ b/doc/source/BuildingRunningTesting/Container.rst @@ -232,8 +232,8 @@ Run the ``setup_container.sh`` script with the proper arguments. Ensure ``LANDDA where: - * ``-c`` is the compiler on the user's local machine (e.g., ``intel/2022.1.2``) - * ``-m`` is the :term:`MPI` on the user's local machine (e.g., ``impi/2022.1.2``) + * ``-c`` is the compiler on the user's local machine (e.g., ``intel/2022.1.2``, ``intel-oneapi-compilers/2022.2.1``, ``intel/2023.2.0``) + * ``-m`` is the :term:`MPI` on the user's local machine (e.g., ``impi/2022.1.2``, ``intel-oneapi-mpi/2021.7.1``, ``cray-mpich/8.1.28``) * ``-i`` is the full path to the container image ( e.g., ``$LANDDAROOT/ubuntu22.04-intel-landda-release-public-v2.0.0.img``). When using a Singularity container, Intel compilers and Intel :term:`MPI` (preferably 2020 versions or newer) need to be available on the host system to properly launch MPI jobs. Generally, this is accomplished by loading a module with a recent Intel compiler and then loading the corresponding Intel MPI. From 13d70af37202d2ed2dc6ed0ac7817f94f5d642e7 Mon Sep 17 00:00:00 2001 From: EdwardSnyder-NOAA <96196752+EdwardSnyder-NOAA@users.noreply.github.com> Date: Fri, 10 Jan 2025 09:42:12 -0600 Subject: [PATCH 3/8] Update Container.rst Add land da xml modification for Gaea --- doc/source/BuildingRunningTesting/Container.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/doc/source/BuildingRunningTesting/Container.rst b/doc/source/BuildingRunningTesting/Container.rst index e5a3c312..15d86b2f 100644 --- a/doc/source/BuildingRunningTesting/Container.rst +++ b/doc/source/BuildingRunningTesting/Container.rst @@ -269,6 +269,14 @@ Once everything looks good, run the `uwtools batch| --clusters=c5 --partition=batch --export=NONE|g' land_analysis.xml + .. _RunExptC: Run the Experiment From 86f0d96ac0c031b391b35680f4e2881c4e9f3d4d Mon Sep 17 00:00:00 2001 From: EdwardSnyder-NOAA <96196752+EdwardSnyder-NOAA@users.noreply.github.com> Date: Fri, 10 Jan 2025 10:17:21 -0600 Subject: [PATCH 4/8] Update Container.rst reformatted the section --- .../BuildingRunningTesting/Container.rst | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/doc/source/BuildingRunningTesting/Container.rst b/doc/source/BuildingRunningTesting/Container.rst index 15d86b2f..6cc462ca 100644 --- a/doc/source/BuildingRunningTesting/Container.rst +++ b/doc/source/BuildingRunningTesting/Container.rst @@ -245,7 +245,23 @@ Configure the Experiment The user should now see the ``Land-DA_workflow`` and ``jedi-bundle`` directories in the ``$LANDDAROOT`` directory. -Because of a conda conflict between the container and the host system, it is best to load rocoto separately instead of using workflow files found in the ``modulefiles`` directory. Note that some modules need to be loaded or a path appended before running the following command. Check the ``modulefiles/wflow_.lua`` to see if that is the case. For Gaea, this path needs to be loaded first ``/ncrc/proj/epic/rocoto/modulefiles/``. +Because of a conda conflict between the container and the host system, it is best to load rocoto separately instead of using workflow files found in the ``modulefiles`` directory. + +.. note:: + + Some modules need to be loaded or a path appended before loading rocoto. + + For Hercules/Orion, add the contrib module before loading rocoto by: + + .. code-block:: console + + module load contrib + + For Gaea, source the following path before loading rocoto by: + + .. code-block:: console + + module use /ncrc/proj/epic/rocoto/modulefiles/ .. code-block:: console From f5e664cef1411145bbc889b13a5f6854de1c3e59 Mon Sep 17 00:00:00 2001 From: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com> Date: Fri, 24 Jan 2025 16:06:58 -0500 Subject: [PATCH 5/8] minor wording edit --- doc/source/BuildingRunningTesting/Container.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/BuildingRunningTesting/Container.rst b/doc/source/BuildingRunningTesting/Container.rst index 6cc462ca..c97e9bce 100644 --- a/doc/source/BuildingRunningTesting/Container.rst +++ b/doc/source/BuildingRunningTesting/Container.rst @@ -249,7 +249,7 @@ Because of a conda conflict between the container and the host system, it is bes .. note:: - Some modules need to be loaded or a path appended before loading rocoto. + On certain systems, some modules need to be loaded or a path appended before loading rocoto. For Hercules/Orion, add the contrib module before loading rocoto by: From 70b3dd95e64ea54710c56353328aab0f542e3a6a Mon Sep 17 00:00:00 2001 From: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com> Date: Fri, 24 Jan 2025 16:07:15 -0500 Subject: [PATCH 6/8] minor typo --- doc/source/BuildingRunningTesting/Container.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/BuildingRunningTesting/Container.rst b/doc/source/BuildingRunningTesting/Container.rst index c97e9bce..e8211b18 100644 --- a/doc/source/BuildingRunningTesting/Container.rst +++ b/doc/source/BuildingRunningTesting/Container.rst @@ -287,7 +287,7 @@ A successful run of these commands will output a “0 errors found” message. .. note:: - To run the container on Gaea, the SLURM options in the ``land_analysis.xml`` needs to be modified by running the following command before starting the experiment: + To run the container on Gaea, the SLURM options in the ``land_analysis.xml`` need to be modified by running the following command before starting the experiment: .. code-block:: console From 5c6cff2cd8bfd5179791ec863276ec5dcf93e371 Mon Sep 17 00:00:00 2001 From: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com> Date: Fri, 24 Jan 2025 16:16:25 -0500 Subject: [PATCH 7/8] Update container info for Orion/Hercules --- .../BuildingRunningTesting/Container.rst | 22 ++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/doc/source/BuildingRunningTesting/Container.rst b/doc/source/BuildingRunningTesting/Container.rst index e8211b18..909db8ac 100644 --- a/doc/source/BuildingRunningTesting/Container.rst +++ b/doc/source/BuildingRunningTesting/Container.rst @@ -85,9 +85,6 @@ Set a top-level directory location for Land DA work, and navigate to it. For exa where ``/path/to/landda`` is the path to this top-level directory (e.g., ``/Users/Joe.Schmoe/landda``). -.. hint:: - If a ``singularity: command not found`` error message appears in any of the following steps, try running: ``module load singularity`` or (on Derecho) ``module load apptainer``. - NOAA RDHPCS Systems ---------------------- @@ -185,6 +182,9 @@ To run the container, users must: Set Up the Container ======================= +.. hint:: + If a ``singularity: command not found`` error message appears in any of the following steps, try running: ``module load singularity`` or (on Derecho) ``module load apptainer``. + Save the location of the container in an environment variable. .. code-block:: console @@ -276,6 +276,14 @@ The ``setup_container.sh`` script creates the ``parm_xml.yaml`` from the ``parm_ Save and close the file. +.. note:: + + On Orion/Hercules, it is also necessary to update the ``SINGULARITYBIN`` variable in ``run_container_executable.sh``: + + .. code-block:: console + + SINGULARITYBIN=/apps/spack-managed/gcc-11.3.1/singularity-3.8.7-ks32erwgzkuf52swkxb5pyzeapwz3i7n/bin/singularity + Once everything looks good, run the `uwtools `_ scripts to create the Rocoto XML file: .. code-block:: console @@ -293,6 +301,14 @@ A successful run of these commands will output a “0 errors found” message. sed -i 's|batch| --clusters=c5 --partition=batch --export=NONE|g' land_analysis.xml +.. note:: + + On some systems, including Orion, users will need to manually export the path to the Python environment by running: + + .. code-block:: console + + export PATH=$LANDDAROOT/land-DA_workflow/sorc/conda/envs/land_da/bin:$PATH + .. _RunExptC: Run the Experiment From 471404c6368f8b4ebc90ed1979e9061b25894bba Mon Sep 17 00:00:00 2001 From: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com> Date: Fri, 24 Jan 2025 17:35:12 -0500 Subject: [PATCH 8/8] Add PATH export for Hercules, too --- doc/source/BuildingRunningTesting/Container.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/BuildingRunningTesting/Container.rst b/doc/source/BuildingRunningTesting/Container.rst index 909db8ac..7474f348 100644 --- a/doc/source/BuildingRunningTesting/Container.rst +++ b/doc/source/BuildingRunningTesting/Container.rst @@ -303,7 +303,7 @@ A successful run of these commands will output a “0 errors found” message. .. note:: - On some systems, including Orion, users will need to manually export the path to the Python environment by running: + On some systems, including Orion/Hercules, users will need to manually export the path to the Python environment by running: .. code-block:: console