Spack Working Group @ SDSC - Meeting - 2023/04/07

[mkandes]

Attendance

• Marty Kandes
• Mahidahr Tatineni
• Subhashini Sivagnanam
• Jerry Greenberg
• Nicole Wolter
• Yuwu Chen
• Trevor Cooper

Agenda

1. Upcoming Planned Maintenance on Expanse

Expected date of maintenance: May 8th, 2023

• Update operating system from Rocky 8.5 to Rocky 8.7
• Update Infiniband driver from Mellanox OFED 4.9-4.1.7 to Mellanox OFED 5.8-1.1.2
• Update Lustre client from lfs 2.12.8_6_g5457c37 to lfs 2.15.2
• Update NVIDIA GPU driver from NVIDIA 510.39.01 CUDA 11.6 to NVIDIA 515.65.01 CUDA 11.7

expanse/0.17.3/cpu and expanse/0.17.3/gpu instances have been modified where necessary and redeployed on the Expanse development (devexp) system or are still in the process of being deployed there. Benchmark have already been run against both instances successfully, including the OSU Micro-Benchmarks (OMB) point-to-point bandwidth and latency tests, GROMACS, and LAMMPS.

Changes to these instances --- expanse/0.17.3/cpu/b and expanse/0.17.3/gpu/b --- will be committed and pushed back to the sdsc/spack repo's sdsc-0.17.3 deployment branch when ready as the pre-production build nodes on Expanse have been repurposed by the HPC Systems Group to deploy their updates to a few nodes for us to then rebuild the new Spack instance against them and provide early user build and testing prior to May 8th maintenance.

2. Update on pre-production testing of expanse/0.17.3/cpu/a and expanse/0.17.3/gpu/a

Ready to close:

• SDSC: PKG - expanse/0.17.3/cpu/a - Missing VASP (example application) #45
• SDSC: PKG - expanse/0.17.3/cpu/a - Missing Trinity (example application) #44
• SDSC: PKG - expanse/0.17.3/cpu/a - Missing ORCA (example application) #40

Ready for testing:

• SDSC: PKG - expanse/0.17.3/cpu/b - Missing matlab (example application) #39
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing gamess (example application) #36
• SDSC: PKG - expanse/0.17.3/cpu/a - Missing cp2k (example application) #35

Open build problems:

• SDSC: PKG - expanse/0.17.3/cpu/b - Missing gaussian (example application) #37
• SDSC: PKG - expanse/0.17.3/gpu/b - Missing Amber GPU(example application) #50

Still need to build test:

• SDSC: PKG - expanse/0.17.3/cpu/b - Missing Wannier90 (example application) #46
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing Abinit (example application) #33
• SDSC: PKG - expanse/0.17.3/cpu/b - Missing Abaqus 2018, 2021 (example application) #32

Update from @nwolter. Follow-up on expanse/0.17.3/gpu/a testing and issue creation.

3. Planning and Preparation for Deployment of Spack on TSCC2

Expected date of deployment: Summer 2023

• Operating system at deployment will be Rocky 9.x.
• Default GCC compiler at for Rocky 9.x is 11.2.x.

Update from @jerrypgreenberg.

4. Upcoming work and/or development issues

• Fix ordering of builds in dependency chains; lmod appears to set the last installed variant of a package as the default
• Restart build out of sdsc/spack spack-0.19.3 deployment branch for Expanse and TSSC2
• Investigate creating a build cache on Jetstream2; may allow us to deploy more quickly during system maintenance
• Complete initial draft of DEPLOYMENT.md, which will serve as general deployment documentation
• Complete initial draft of CONTRIBUTING.md, which will serve as documentation on how to contribute work to this repo
• Create a dev & test spack instance(s) based on a subset of the the expanse/0.17.3/cpu and expanse/0.17.3/gpu instances

References:

• https://hpc-sdsc.atlassian.net/wiki/spaces/HO/pages/8424125/Expanse+Acceptance+Testing
• https://hpc-sdsc.atlassian.net/wiki/spaces/HO/pages/8423521/Expanse+Spack+Application+Deployment

[mkandes]

Revisit knem, xpmem, and other MPI-related system packages that may improve performance as external packages.

[mkandes]

Check if @tcooper has installed gdr-related driver and kernel modules have been put in new Expanse images.

[mkandes]

@mkandes - Check for spack command that sets lmod defaults and then update spec build scripts.