Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge release/1.4.1 into develop #671

Merged
merged 19 commits into from
Jul 14, 2023
Merged

Merge release/1.4.1 into develop #671

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
b517acf
set build_type=Release for fms
AlexanderRichert-NOAA Jun 20, 2023
ef0f3d7
update scotch variants in configs/common/packages.yaml
AlexanderRichert-NOAA Jun 20, 2023
53de65a
update .gitmodules
AlexanderRichert-NOAA Jun 20, 2023
f10f1e3
submodule pointer
AlexanderRichert-NOAA Jun 20, 2023
c3a4140
submodule pointer
AlexanderRichert-NOAA Jun 20, 2023
35298b1
update gitmodules
AlexanderRichert-NOAA Jun 20, 2023
1e1c2a4
submodule pointer
AlexanderRichert-NOAA Jun 20, 2023
2181c7e
remove tab
AlexanderRichert-NOAA Jun 20, 2023
45e96ce
fix gitmodules file
AlexanderRichert-NOAA Jun 20, 2023
928ee77
Update eckit to 1.23.1 (#635)
climbfuji Jun 20, 2023
248628a
Merge branch 'release/1.4.1' of https://github.com/jcsda/spack-stack …
climbfuji Jun 20, 2023
3322236
Merge pull request #632 from AlexanderRichert-NOAA/ufs_jun2023
AlexanderRichert-NOAA Jun 20, 2023
17e75be
Update Acorn config and include [email protected] for 1.4.1 (#636)
AlexanderRichert-NOAA Jun 21, 2023
6ffd4e7
update container README (#639)
AlexanderRichert-NOAA Jun 21, 2023
8475534
update submodule pointer for scotch mpi patch (#660)
AlexanderRichert-NOAA Jul 3, 2023
be8f8dd
Final updates (documentation, site configs, ...) for spack-stack-1.4.…
climbfuji Jul 14, 2023
7e50533
Merge branch 'release/1.4.1' of https://github.com/jcsda/spack-stack …
climbfuji Jul 14, 2023
c429794
Update .gitmodules and submodule pointer for spack for code review an…
climbfuji Jul 14, 2023
93e2a71
[skip ci] Revert .gitmodules and update submodule pointer for spack
climbfuji Jul 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions configs/common/packages.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
version: [1.78.0]
variants: ~atomic +chrono +date_time +exception +filesystem ~graph ~iostreams ~locale ~log ~math ~mpi ~numpy +pic +program_options +python ~random +regex +serialization ~signals +system +test +thread +timer ~wave cxxstd=14 visibility=hidden
bufr:
version: [11.7.1]
version: [12.0.0]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'm good with making [email protected] the default version, but I want to double check. JEDI users should be using spack-stack-1.4.0 which has [email protected] installed. Because of this JEDI users will be insulated from this change so it's okay to make [email protected] the default version on the develop branch.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a note: [email protected] is producing large differences in GSI regression tests (as reported by @DavidHuber-NOAA: NOAA-EMC/GSI#589 (comment)). fine to merge this here, but i expect there might be some issues w/ some applications during transition to this version.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ulmononian one thing to check would be if the GSI is currently using the [email protected] bufr_8 component (which is using 8-byte integers and 8-byte floats, ie double precision), or the bufr_d component (which has 4-byte integers and 8-byte floats). See the README.md notes at https://github.com/NOAA-EMC/NCEPLIBS-bufr/tree/bufr_v11.7.1 for details.

The only component available in [email protected] is bufr_4 which is 4-byte integers and 4-byte floats (ie single precision) so the GSI bufr related code could be making a switch from double to single precision.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me note that if it turns out that we need both bufr@12 and bufr@11 in spack-stack-1.5.0, we can install both and configure the different environments (gsi-env etc.) such that they load the correct one.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@srherbener Indeed, the GSI uses the bufr_d library, so that could explain the differences. Thanks!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@srherbener thanks for providing that insight, steve. @DavidHuber-NOAA do you expect the GSI team to accept the differences w/ bufr_4 / bufr@12 or should we plan to provide bufr@11 for gsi purposes for the foreseeable future?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ulmononian I don't expect that they will accept it or at least that they will want to run some more testing before making the switch. Also, I don't see any hints on their GitHub page that they desire to upgrade at this time. But I have asked the question -- and will ask again on a different channel -- and will update when I hear back.

variants: +python
# Newer versions of CDO require the C++-17 standard, which doesn't
# work with all compilers that are currently in use in spack-stack
Expand All @@ -42,7 +42,7 @@
version: [5.8.4]
variants: +ui
eckit:
version: [1.23.0]
version: [1.23.1]
variants: linalg=eigen,lapack compression=lz4,bzip2
ecmwf-atlas:
version: [0.33.0]
Expand Down Expand Up @@ -70,7 +70,7 @@
version: [1.1.0]
fms:
version: [2023.01]
variants: precision=32,64 +quad_precision +gfs_phys +openmp +pic constants=GFS
variants: precision=32,64 +quad_precision +gfs_phys +openmp +pic constants=GFS build_type=Release
climbfuji marked this conversation as resolved.
Show resolved Hide resolved
g2:
version: [3.4.5]
g2c:
Expand Down Expand Up @@ -248,6 +248,7 @@
version: [5.15.3]
scotch:
version: [7.0.3]
variants: +mpi+metis~shared~threads~mpi_thread+noarch
sfcio:
version: [1.4.1]
shumlib:
Expand Down
12 changes: 6 additions & 6 deletions configs/containers/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@

To avoid hardcoding specs in the generic container recipes, we keep the specs list empty (`specs: []`) and manually add the specs for the particular spack-stack release and application as listed below, *after* running `spack stack create ctr`.

### spack-stack-1.4.0 / skylab-5.0.0 containers for fv3-jedi and mpas-jedi (but not for ufs-jedi)
### spack-stack-1.4.1 / skylab-5.0.0 containers for fv3-jedi and mpas-jedi (but not for ufs-jedi)
```
specs: [[email protected], [email protected] ~fftw, [email protected], [email protected],
[email protected], [email protected], [email protected], bufr@11.7.1, [email protected], [email protected], ecflow@5,
[email protected].0, [email protected] +trans ~fftw, [email protected] ~fftw, [email protected],
[email protected], [email protected], [email protected], bufr@12.0.0, [email protected], [email protected], ecflow@5,
[email protected].1, [email protected] +trans ~fftw, [email protected] ~fftw, [email protected],
[email protected], fms@release-jcsda, [email protected], [email protected], [email protected],
[email protected], [email protected], [email protected], [email protected], [email protected], [email protected],
[email protected], [email protected], [email protected], [email protected],
Expand All @@ -27,14 +27,14 @@ To avoid hardcoding specs in the generic container recipes, we keep the specs li
# [email protected]
```

### spack-stack-1.4.0 / ufs-weather-model-x.y.z containers for ufs-weather-model as of May 18, 2023
### spack-stack-1.4.1 / ufs-weather-model-x.y.z containers for ufs-weather-model as of July 5, 2023

**Note. This is not yet working correctly, some libraries are missing. Please do not use yet! Also, if using the clang-mpich container, need to disable openmp for fms, not clear how to do this cleanly.**

```
specs: [[email protected],
[email protected], [email protected], bufr@11.7.1, [email protected], [email protected], ecflow@5,
[email protected].0, [email protected] +trans ~fftw, [email protected] ~fftw, [email protected],
[email protected], [email protected], bufr@12.0.0, [email protected], [email protected], ecflow@5,
[email protected].1, [email protected] +trans ~fftw, [email protected] ~fftw, [email protected],
[email protected], [email protected], [email protected], [email protected], [email protected],
[email protected], [email protected], [email protected], [email protected], [email protected], [email protected],
[email protected], [email protected], [email protected], [email protected],
Expand Down
6 changes: 3 additions & 3 deletions configs/sites/gaea-c5/packages.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ packages:
buildable: False
externals:
- spec: [email protected]+ui+static_boost
prefix: /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/ecflow-5.8.4-c5
prefix: /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/c5/ecflow-5.8.4
modules: [ecflow/5.8.4]
file:
externals:
Expand Down Expand Up @@ -174,7 +174,7 @@ packages:
buildable: False
externals:
- spec: [email protected]
prefix: /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/mysql-8.0.31-c5
prefix: /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/c5/mysql-8.0.31
modules: [mysql/8.0.31]
ncurses:
externals:
Expand Down Expand Up @@ -204,7 +204,7 @@ packages:
qt:
externals:
- spec: [email protected]
prefix: /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/qt-5.15.2-c5/5.15.2/gcc_64
prefix: /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/c5/qt-5.15.2/5.15.2/gcc_64
rdma-core:
externals:
- spec: [email protected]
Expand Down
4 changes: 2 additions & 2 deletions configs/sites/hercules/mirrors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
mirrors:
local-source:
fetch:
url: file:///work/noaa/da/role-da/spack-stack/source-cache
url: file:///work/noaa/epic/role-epic/spack-stack/source-cache
access_pair:
- null
- null
access_token: null
profile: null
endpoint_url: null
push:
url: file:///work/noaa/da/role-da/spack-stack/source-cache
url: file:///work/noaa/epic/role-epic/spack-stack/source-cache
access_pair:
- null
- null
Expand Down
23 changes: 15 additions & 8 deletions configs/sites/hercules/packages.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ packages:
compiler:: [[email protected], [email protected]]
#compiler:: [[email protected]]
providers:
mpi:: [[email protected], [email protected].4]
mpi:: [[email protected], [email protected].5]

### MPI, Python, MKL
mpi:
Expand All @@ -20,11 +20,11 @@ packages:
# - intel-oneapi-mpi/2021.7.1
openmpi:
externals:
- spec: [email protected].4%[email protected] ~cuda~cxx~cxx_exceptions~java~memchecker+pmi+static~wrapper-rpath
- spec: [email protected].5%[email protected]~cuda~cxx~cxx_exceptions~java~memchecker+pmi~static~wrapper-rpath
fabrics=ucx schedulers=slurm
prefix: /apps/spack-managed/gcc-11.3.1/openmpi-4.1.4-ruvlmb6yyvzbzbiqaov4zk75ogthczsp
prefix: /work/noaa/epic/role-epic/spack-stack/hercules/openmpi-4.1.5/gcc-11.3.1
modules:
- openmpi/4.1.4
- openmpi/4.1.5
python:
buildable: False
externals:
Expand Down Expand Up @@ -74,9 +74,9 @@ packages:
buildable: False
externals:
- spec: [email protected]+ui+static_boost
prefix: /work/noaa/epic-ps/role-epic-ps/spack-stack/ecflow-5.8.4-hercules
prefix: /work/noaa/epic/role-epic/spack-stack/hercules/ecflow-5.8.4
modules:
- ecflow/5.8.4-hercules
- ecflow/5.8.4
findutils:
externals:
- spec: [email protected]
Expand Down Expand Up @@ -119,9 +119,9 @@ packages:
buildable: False
externals:
- spec: [email protected]
prefix: /work/noaa/epic-ps/role-epic-ps/spack-stack/mysql-8.0.31-hercules
prefix: /work/noaa/epic/role-epic/spack-stack/hercules/mysql-8.0.31
modules:
- mysql/8.0.31-hercules
- mysql/8.0.31
openssh:
externals:
- spec: [email protected]
Expand Down Expand Up @@ -164,3 +164,10 @@ packages:
externals:
- spec: [email protected]
prefix: /usr
# Need to use external zlib, because of qt dependence on it (otherwise issues with tar command)
zlib:
externals:
- spec: [email protected]
prefix: /apps/spack-managed/gcc-11.3.1/zlib-1.2.13-ltp4c3zzde3zi3gf7x4b7c7nj5ww4i4g
modules:
- zlib/1.2.13
4 changes: 2 additions & 2 deletions configs/sites/orion/mirrors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
mirrors:
local-source:
fetch:
url: file:///work/noaa/da/role-da/spack-stack/source-cache
url: file:///work/noaa/epic/role-epic/spack-stack/source-cache
access_pair:
- null
- null
access_token: null
profile: null
endpoint_url: null
push:
url: file:///work/noaa/da/role-da/spack-stack/source-cache
url: file:///work/noaa/epic/role-epic/spack-stack/source-cache
access_pair:
- null
- null
Expand Down
2 changes: 1 addition & 1 deletion doc/modulefile_templates/openmpi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ unsetenv SLURM_EXPORT_ENV
setenv PSM2_PATH_SELECTION "static_base"
setenv SLURM_CPU_BIND "none"

# Settings specific for Cheyenne
# Settings specific for Cheyenne and Hercules
setenv MPI_ROOT ${OPENMPI_PATH}
setenv UCX_MAX_RNDV_RAILS "1"
setenv OMPI_MCA_btl "^openib"
38 changes: 24 additions & 14 deletions doc/source/MaintainersSection.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@ Sign into qt, select customized installation, choose [email protected] only (uncheck all
.. note::
On air-gapped systems, the above method may not work (we have not encountered such a system so far).

.. note::
If ``./qt-unified-linux-x64-online.run`` fails to start with the error ``qt.qpa.xcb: could not connect to display`` and a role account is being used, follow the procedure described in https://www.thegeekdiary.com/how-to-set-x11-forwarding-export-remote-display-for-users-who-switch-accounts-using-sudo to export the display. A possible warning ``xauth: file /ncrc/home1/role.epic/.Xauthority does not exist`` can be ignored, since this file gets created by the ``xauth`` command.

.. _MaintainersSection_ecFlow:

------------------------------
Expand Down Expand Up @@ -243,7 +246,7 @@ MSU Hercules
------------------------------

ecflow
``ecFlow`` must be built manually using the GNU compilers and linked against a static ``boost`` library, using an available ``Qt5`` installation. After loading the following modules, follow the instructions in :numref:`Section %s <MaintainersSection_ecFlow>` to install ``ecflow`` in ``/work/noaa/epic-ps/role-epic-ps/spack-stack/ecflow-5.8.4-hercules``.
``ecFlow`` must be built manually using the GNU compilers and linked against a static ``boost`` library, using an available ``Qt5`` installation. After loading the following modules, follow the instructions in :numref:`Section %s <MaintainersSection_ecFlow>` to install ``ecflow`` in ``/work/noaa/epic/role-epic/spack-stack/hercules/ecflow-5.8.4``.

.. code-block:: console

Expand All @@ -253,6 +256,22 @@ ecflow
mysql
``mysql`` must be installed separately from ``spack`` using a binary tarball provided by the MySQL community. Follow the instructions in :numref:`Section %s <MaintainersSection_MySQL>` to install ``mysql`` in ``/work/noaa/epic-ps/role-epic-ps/spack-stack/mysql-8.0.31-hercules``.

openmpi
need to load qt so to get consistent zlib (or just load zlib directly, check qt module)

.. code-block:: console

module purge
module load zlib/1.2.13
module load ucx/1.13.1
./configure \
--prefix=/work/noaa/epic/role-epic/spack-stack/hercules/openmpi-4.1.5/gcc-11.3.1 \
--with-ucx=$UCX_ROOT \
--with-zlib=$ZLIB_ROOT
make VERBOSE=1 -j4
make check
make install

.. _MaintainersSection_Discover:

------------------------------
Expand Down Expand Up @@ -538,17 +557,8 @@ NOAA RDHPCS Gaea C5

On Gaea C5, ``miniconda``, ``qt``, ``ecflow``, and ``mysql`` need to be installed as a one-off before spack can be used.

miniconda
Follow the instructions in :numref:`Section %s <MaintainersSection_Miniconda>` to create a basic ``miniconda`` installation and associated modulefile for working with spack. Don't forget to log off and back on to forget about the conda environment. Use the following workaround to avoid the terminal being spammed by error messages about missing version information (``/usr/bin/lua5.3: /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/miniconda-3.9.12-c5/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)``):

.. code-block:: console

cd /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/miniconda-3.9.12-c5/lib
mv libtinfow.so.6.3 libtinfow.so.6.3.conda.original
ln -sf /lib64/libtinfo.so.6 libtinfow.so.6.3

qt (qt@5)
The default ``qt@5`` in ``/usr`` is incomplete and thus insufficient for building ``ecflow``. After loading/unloading the modules as shown below, refer to :numref:`Section %s <MaintainersSection_Qt5>` to install ``[email protected]`` in ``/lustre/f2/dev/wpo/role.epic/contrib/spack-stack/qt-5.15.2-c5``. Note that the installation must be done as a regular user due to problems with graphical applications for role accounts.
The default ``qt@5`` in ``/usr`` is incomplete and thus insufficient for building ``ecflow``. After loading/unloading the modules as shown below, refer to :numref:`Section %s <MaintainersSection_Qt5>` to install ``[email protected]`` in ``/lustre/f2/dev/wpo/role.epic/contrib/spack-stack/c5/qt-5.15.2``. :numref:`Section %s <MaintainersSection_Qt5>` describes how to export the X windows environment in order to install ``qt@5`` using the role account.

.. code-block:: console

Expand All @@ -557,7 +567,7 @@ qt (qt@5)
module load PrgEnv-gnu/8.3.3

ecflow
``ecFlow`` must be built manually using the GNU compilers and linked against a static ``boost`` library. After installing `qt5` and loading the following modules, follow the instructions in :numref:`Section %s <MaintainersSection_ecFlow>`. Because of the dependency on ``miniconda``, that module must be loaded automatically in the ``ecflow`` module (similar to ``[email protected]``). Ensure to follow the extra instructions in that section for Gaea C5.
``ecFlow`` must be built manually using the GNU compilers and linked against a static ``boost`` library. After installing `qt5` and loading the following modules, follow the instructions in :numref:`Section %s <MaintainersSection_ecFlow>`. Because of the dependency on ``miniconda``, that module must be loaded automatically in the ``ecflow`` module (similar to ``[email protected]``). Ensure to follow the extra instructions in that section for Gaea C5 in ``/lustre/f2/dev/wpo/role.epic/contrib/spack-stack/c5/ecflow-5.8.4``.

Ensure to follow the extra instructions in that section for Gaea.

Expand All @@ -568,11 +578,11 @@ ecflow
module load PrgEnv-gnu/8.3.3
module load python/3.9.12

module use /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/modulefiles-c5
module use /lustre/f2/dev/wpo/role.epic/contrib/spack-stack/c5/modulefiles
module load qt/5.15.2

mysql
``mysql`` must be installed separately from ``spack`` using a binary tarball provided by the MySQL community. Follow the instructions in :numref:`Section %s <MaintainersSection_MySQL>` to install ``mysql`` in ``/lustre/f2/dev/wpo/role.epic/contrib/spack-stack/mysql-8.0.31-c5``.
``mysql`` must be installed separately from ``spack`` using a binary tarball provided by the MySQL community. Follow the instructions in :numref:`Section %s <MaintainersSection_MySQL>` to install ``mysql`` in ``/lustre/f2/dev/wpo/role.epic/contrib/spack-stack/c5/mysql-8.0.31``.

.. _MaintainersSection_Hera:

Expand Down
Loading