Spack Working Group @ SDSC - Meeting - 2023/04/28

[mkandes]

Attendance

• Marty Kandes
• Mahidahr Tatineni
• Subhashini Sivagnanam
• Jerry Greenberg
• Nicole Wolter
• Yuwu Chen
• Trevor Cooper

Agenda

1. Announcements

Upcoming vacation(s):

• Marty: Thursday, May, 11th - Monday, May 22nd

Next Meeting(s):

• Friday, May 5th
• Friday, May 26th

Propose moving to bi-weekly meetings starting from Friday, May 26th, unless we determine at that time more frequent, weekly meetings are still needed to assist with TSSC2 Spack preparation, deployment, and testing.

2. Upcoming Planned Maintenance on Expanse

Expected date of maintenance: Monday, May 8th, 2023

• Update operating system from Rocky 8.5 to Rocky 8.7
• Update Infiniband driver from Mellanox OFED 4.9-4.1.7 to Mellanox OFED 5.8-1.1.2
• Update Lustre client from lfs 2.12.8_6_g5457c37 to lfs 2.15.2
• Update NVIDIA GPU driver from NVIDIA 510.39.01 CUDA 11.6 to NVIDIA 515.65.01 CUDA 11.7

The initial deployments of both new production Spack instances expanse/0.17.3/cpu/b and expanse/0.17.3/gpu/b are now in place on Expanse.

Re-running of standard benchmark tests is ongoing.

The entire record of the deployment for expanse/0.17.3/cpu/b has been committed back to the sdsc/spack repo, including the standard output files from the build jobs for future reference.

When tested during deployment, it appears the spack module lmod setdefault is not sticky. i.e., the default module appear to need to be set post-deployment. This has not yet been done.

3. Update on pre-production testing of expanse/0.17.3/cpu/a and expanse/0.17.3/gpu/a

Ready to close:

• None

Ready for testing:

• SDSC: PKG - expanse/0.17.3/cpu/b - Missing matlab (example application) #39
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing gamess (example application) #36
• SDSC: PKG - expanse/0.17.3/cpu/a - Missing cp2k (example application) #35
• SDSC: PKG - expanse/0.17.3/cpu/a - Missing QE (example application) #63
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing NWCHEM (example application) #62
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing NEURON (example application) #61
• SDSC: PKG - expanse/0.17.3/cpu/a - Missing NAMD (example application) #60
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing gromacs(example application) #58
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing LAMMPS (example application) #55
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing gromacs GPU(example application) #54

Open runtime problems:

• ATS-2095/ZEN-30304 - expanse/0.17.3/gpu/b - Missing LAMMPS GPU (example application) #56

Open build problems:

• SDSC: PKG - expanse/0.17.3/cpu/b - Missing gaussian (example application) #37
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing Amber GPU(example application) #50
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing NAMD GPU (example application) #59
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing gaussian (example application) #57

Still need to build test:

• SDSC: PKG - expanse/0.17.3/cpu/b - Missing Wannier90 (example application) #46
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing Abinit (example application) #33
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing Abaqus 2018, 2021 (example application) #32
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing matlab (example application) #67
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing Amber GPU(example application) #50

Updates from @nwolter.

4. Planning and Preparation for Deployment of Spack on TSCC2

Expected date of deployment: Summer 2023

• Operating system at deployment will be Rocky 9.x
• Default GCC for Rocky 9.x is gcc 11.2.x.
• All networking will be RoCE.
• There will be a Lustre filesystem.

Updates from @jerrypgreenberg.

Open questions:

• Does TSSC2 have spack user role accounts for deployment?
• Is TSSC2 ready for write access to /cm/shared?

5. Upcoming work and/or development issues

• Complete initial draft of DEPLOYMENT.md, which will serve as general deployment documentation
• Complete initial draft of CONTRIBUTING.md, which will serve as documentation on how to contribute work to this repo
• Spack v0.19.2 is out; https://github.com/spack/spack/releases/tag/v0.19.2
• Create and start build out of a sdsc/spack spack-0.19.2 deployment branch for Expanse and TSSC2
• Create a dev & test spack instance(s) based on a subset of the the expanse/0.17.3/cpu and expanse/0.17.3/gpu instances
• Investigate creating a build cache on Jetstream2; may allow us to deploy more quickly during system maintenance

References:

• https://hpc-sdsc.atlassian.net/wiki/spaces/HO/pages/8424125/Expanse+Acceptance+Testing
• https://hpc-sdsc.atlassian.net/wiki/spaces/HO/pages/8423521/Expanse+Spack+Application+Deployment

[mkandes]

@tcooper wants us to run some tests on exp-11-04 as it has the kernel update for the cve.

[mkandes]

@mkandes - namd issue for expanse/0.17.3/cpu/a can likely be closed

[mkandes]

@jerrypgreenberg - mentioned lua logging.

[mkandes]

https://lmod.readthedocs.io/en/latest/300_tracking_module_usage.html