Initial software set #44

heatherkellyucl · 2023-07-19T12:18:43Z

Large usage MD codes

LAMMPS + variants: Install Request: LAMMPS with Python in version of 7 Feb 2024 or later UCL-RITS/rcps-buildscripts#575
CP2K: Install Request: CP2K v2023.1 (latest version) [IN05824831] UCL-RITS/rcps-buildscripts#511
GROMACS + variants: Install Request: GROMACS 2023 (latest update) including CPU, GPU and + Plumed versions [IN05756873] [IN05807314][IN05844418] UCL-RITS/rcps-buildscripts#508 (plumed only available as variant for 2023.x, plumed for 2024 is not ready yet)
CASTEP: Install Request: CASTEP 2x [IN05839181] UCL-RITS/rcps-buildscripts#514 Pull request for initial spec at Create Castep package spack/spack#41230 - is now merged.
ORCA (binary, but needs a working OpenMPI 4.1.1): Install Request: ORCA 5 UCL-RITS/rcps-buildscripts#474
NAMD
Check whether we want to put VASP in atm or leave it out (lots of use, but we have a recent install).

Myriad-flavoured packages

[Initial stack done #49]
Samtools, BCFTools, HTSlib: UCL-RITS/rcps-buildscripts#532

Samtools, Bwa, Bedtools, Vcftools, picard, GATK (requested by Biosciences) https://ucldata.atlassian.net/browse/AHS-139

Alphafold (would be nice to not have to tell people to go and track down the working containers): UCL-RITS/rcps-buildscripts#529 UCL-RITS/rcps-buildscripts#463 [Not doing until Spack 0.22]

HDF5
Netcdf
BEAST 2.7: UCL-RITS/rcps-buildscripts#498

The list of user-requested software builds is at https://github.com/UCL-RITS/rcps-buildscripts/issues

Second set

Amber
Cellranger: UCL-RITS/rcps-buildscripts#499
Have a look at dependencies of SRAtoolkit: UCL-RITS/rcps-buildscripts#543
Hammock
AFNI

Out of scope for initial set

Python
R

GPU builds

Only Myriad and Young have GPUs. Young's GPUs are AMD-cpu nodes while Myriad's are Intel. (GPUs themselves are Nvidia). Young only has A100s, Myriad has A100s and some older GPUs, of which the oldest will be retired soonish.

GPU builds are dependent on the version of the CUDA drivers being updated on the GPU nodes - not worth putting any GPU builds in the buildcache until after that has happened. UCL-RITS/rcps-buildscripts#528 and UCL-RITS/rcps-buildscripts#517 and https://ucldata.atlassian.net/browse/ARI-254. Done.

Work cannot begin on Kathleen and Michael until https://ucldata.atlassian.net/browse/ARI-37 is complete (including devtoolset-11 and rh-python38 in the image). Done.

heatherkellyucl · 2023-07-19T12:49:27Z

FSL in Spack patches out the job submission capability.
Freesurfer doesn't depend on Matlab but uses it.

heatherkellyucl · 2023-07-19T12:51:29Z

We were considering compilers but Spack is about to change how it deals with them so probably worth waiting.

heatherkellyucl · 2023-07-25T13:43:07Z

We want to test that the OpenMPI we get from Spack is reasonable performance-wise on the OmniPath clusters.

So far we have not specified any fabrics and Spack lets OpenMPI set options as auto in that case.

We should check performance of that version against one specifying some likely options (and can test against the vader-using mpi/openmpi/4.1.1/gnu-4.9.2 too - if it performs worse than that, we have a problem!)

Benchmarks: https://mvapich.cse.ohio-state.edu/benchmarks/

Can see available variants with spack info --all openmpi

krishnakumarg1984 · 2023-08-25T09:43:30Z

Everything is working finally, and have been going fine for the last 2 days.

Currently, building [email protected]%[email protected] ^[email protected] schedulers=sge %[email protected].

The complete provenance is:

[email protected]%[email protected]~blas~cp2k~cuda~cycle_subcounters~double+hwloc~ipo~lapack~mdrun_only+mpi~nosuffix~opencl+openmp~relaxed_double_precision+shared~sycl build_system=cmake build_type=Release generator=make openmp_max_threads=none arch=linux-rhel7-broadwell
    ^[email protected]%[email protected]~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-rhel7-broadwell
        ^[email protected]%[email protected]~symlinks+termlib abi=none build_system=autotools arch=linux-rhel7-broadwell
        ^[email protected]%[email protected]~docs~shared build_system=generic certs=mozilla arch=linux-rhel7-broadwell
            ^ca-certificates-mozilla@2023-01-10%[email protected] build_system=generic arch=linux-rhel7-broadwell
    ^[email protected]%[email protected]+mpi~openmp~pfft_patches build_system=autotools precision=double,float arch=linux-rhel7-broadwell
    ^[email protected]%[email protected]~guile build_system=autotools arch=linux-rhel7-broadwell
    ^[email protected]%[email protected]~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-rhel7-broadwell
        ^[email protected]%[email protected] build_system=autotools arch=linux-rhel7-broadwell
            ^[email protected]%[email protected] build_system=autotools arch=linux-rhel7-broadwell
            ^[email protected]%[email protected] build_system=autotools arch=linux-rhel7-broadwell
        ^[email protected]%[email protected]~python build_system=autotools arch=linux-rhel7-broadwell
            ^[email protected]%[email protected] build_system=autotools libs=shared,static arch=linux-rhel7-broadwell
            ^[email protected]%[email protected]~pic build_system=autotools libs=shared,static arch=linux-rhel7-broadwell
        ^[email protected]%[email protected] build_system=autotools arch=linux-rhel7-broadwell
    ^[email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~orterunprefix+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools fabrics=none schedulers=sge arch=linux-rhel7-broadwell
        ^[email protected]%[email protected] build_system=autotools patches=4e1d78c,62fc8a8,ff37630 arch=linux-rhel7-broadwell
            ^[email protected]%[email protected] build_system=autotools patches=35c4492,7793209,a49dd5b arch=linux-rhel7-broadwell
            ^[email protected]%[email protected] build_system=autotools arch=linux-rhel7-broadwell
            ^[email protected]%[email protected]+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rhel7-broadwell
                ^[email protected]%[email protected] build_system=autotools arch=linux-rhel7-broadwell
                ^[email protected]%[email protected] build_system=autotools arch=linux-rhel7-broadwell
        ^[email protected]%[email protected]+gssapi build_system=autotools arch=linux-rhel7-broadwell
            ^[email protected]%[email protected]+shared build_system=autotools arch=linux-rhel7-broadwell
                ^[email protected]%[email protected] build_system=autotools arch=linux-rhel7-broadwell
                ^[email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-rhel7-broadwell
                    ^[email protected]%[email protected] build_system=autotools zip=pigz arch=linux-rhel7-broadwell
                        ^[email protected]%[email protected] build_system=makefile arch=linux-rhel7-broadwell
                        ^[email protected]%[email protected]+programs build_system=makefile compression=none libs=shared,static arch=linux-rhel7-broadwell
            ^[email protected]%[email protected] build_system=autotools arch=linux-rhel7-broadwell
            ^[email protected]%[email protected]~obsolete_api build_system=autotools arch=linux-rhel7-broadwell
        ^[email protected]%[email protected]+cpanm+open+shared+threads build_system=generic arch=linux-rhel7-broadwell
            ^[email protected]%[email protected]+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rhel7-broadwell
            ^[email protected]%[email protected]~debug~pic+shared build_system=generic arch=linux-rhel7-broadwell
            ^[email protected]%[email protected] build_system=autotools arch=linux-rhel7-broadwell
                ^[email protected]%[email protected] build_system=autotools patches=bbf97f1 arch=linux-rhel7-broadwell
        ^[email protected]%[email protected]~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=linux-rhel7-broadwell
            ^[email protected]%[email protected]+openssl build_system=autotools arch=linux-rhel7-broadwell
        ^[email protected]%[email protected]+optimize+pic+shared build_system=makefile arch=linux-rhel7-broadwell

heatherkellyucl · 2023-08-29T12:15:56Z

Looks like the gromacs variants we would have, based on our current ones, are these. Or for 2022.5 we could just do the bolded plumed variants as we can't build those yet for 2023.

We normally include the double-precision versions with the non-cuda builds.

cuda_arch=80 is the A100s. Ignore the cuda variants for the first go-round. (Would only be added on Myriad and Young, and on Young it'd need to be built on a GPU node as those are AMD).

[email protected] %[email protected] +double
[email protected] %[email protected] +double +plumed
[email protected] %[email protected] +cuda cuda_arch=80
[email protected] %[email protected] +cuda cuda_arch=80 +plumed

[email protected] %[email protected] +double
[email protected] %[email protected] +cuda cuda_arch=80

krishnakumarg1984 · 2023-08-29T12:55:15Z

Ok. Happy to do just the bolded ones for now.

heatherkellyucl · 2023-08-29T13:59:31Z

Not just the bolded ones, [email protected] %[email protected] +double is fine, and should be related to what you were already doing, just with +double.

heatherkellyucl · 2023-09-01T13:52:45Z

@krishnakumarg1984 When you have installed some packages, spack find -ldf will show you a tree.

kut6cpb [email protected]%gcc
y5z2x63     [email protected]%gcc
rrab7ki         [email protected]%gcc
ehpjj76     [email protected]%gcc
uaerdna         [email protected]%gcc
pokguyu     [email protected]%gcc
qw2elyk         [email protected]%gcc
jtrpaef         [email protected]%gcc
3yevelt     [email protected]%gcc
scvmoox     [email protected]%gcc
gphegdo         [email protected]%gcc
3fsyut7         [email protected]%gcc
wiuk7k4         [email protected]%gcc
5a6s7ss     [email protected]%gcc
t6o5yxx     [email protected]%gcc
43lsbjo     [email protected]%gcc
rmhkp3h         [email protected]%gcc
che7ezb     [email protected]%gcc
6pqhjod         [email protected]%gcc
mq5csuf         [email protected]%gcc
afgghmq         [email protected]%gcc
5xsqgvu     [email protected]%gcc
rp5rvar         [email protected]%gcc
lfr5oag             [email protected]%gcc
sqh3ckt             [email protected]%gcc
omwksxg                 [email protected]%gcc
7ibfohu             [email protected]%gcc
6nrbosy         [email protected]%gcc
gsyujwx     [email protected]%gcc
myexac2     [email protected]%gcc

afgghmq [email protected]%gcc
jtrpaef     [email protected]%gcc
6nrbosy         [email protected]%gcc
rrab7ki             [email protected]%gcc

etc since you'll get one for every individual package.

heatherkellyucl · 2023-09-01T17:18:48Z

I'm pretty sure it says this somewhere else in here, but for a site called envtest and an environment called compilers, then these are created during concretisation and list the specs and hashes for that environment. (And the concretization is shown on the command line as it does it).

$sites_root/envtest/spack/var/spack/environments/compilers/spack.yaml
$sites_root/envtest/spack/var/spack/environments/compilers/spack.lock

There's a .spack-env in there too. We're creating an environment in our site: https://spack.readthedocs.io/en/latest/environments.html

krishnakumarg1984 · 2023-09-12T14:45:05Z

@heatherkellyucl Done. Finally got it working.

[+] /scratch/home/uccagop/apps/spack/0.20/krishna_spacksite_michael_ucl/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-12.2.0/fftw-3.3.10-3y2q7r6hhiqhzofiawmfxdmem72y2ikp
==> Installing gromacs-2023.1-iqsepjm7yg5q2dpbfbr622xvwph4rgxu
==> No binary for gromacs-2023.1-iqsepjm7yg5q2dpbfbr622xvwph4rgxu found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/ee/eef2bb4a6cb6314cf9da47f26df2a0d27af4bf7b3099723d43601073ab0a42f4.tar.gz
==> Ran patch() for gromacs
==> gromacs: Executing phase: 'cmake'
==> gromacs: Executing phase: 'build'
==> gromacs: Executing phase: 'install'
==> gromacs: Successfully installed gromacs-2023.1-iqsepjm7yg5q2dpbfbr622xvwph4rgxu
  Stage: 24.40s.  Cmake: 1m 9.11s.  Build: 8m 5.01s.  Install: 10.59s.  Post-install: 3.34s.  Total: 9m 53.94s
[+] /scratch/home/uccagop/apps/spack/0.20/krishna_spacksite_michael_ucl/spack/opt/spack/[padded-to-256-chars]/linux-rhel7-broadwell/gcc-12.2.0/gromacs-2023.1-iqsepjm7yg5q2dpbfbr622xvwph4rgxu
==> Updating view at /scratch/home/uccagop/apps/spack/0.20/krishna_spacksite_michael_ucl/spack/var/spack/environments/krishna_spackenv_michael_ucl/.spack-env/view
[krishna_spackenv_michael_ucl] [uccagop@login11 hpc-spack]$

==> Installed packages
-- linux-rhel7-broadwell / [email protected] ---------------------------
iqsepjm [email protected]
incrgec     [email protected]
6nrbosy         [email protected]
x5spvhy         [email protected]
76n2l7a             ca-certificates-mozilla@2023-01-10
3y2q7r6     [email protected]
dvrmwg4     [email protected]
hmy42iw     [email protected]
ionflx6         [email protected]
5a6s7ss             [email protected]
v26d2xt             [email protected]
lfr5oag         [email protected]
uaerdna             [email protected]
7ibfohu             [email protected]
rrab7ki         [email protected]
mu7mzua     [email protected]
l6vxr43         [email protected]
gphegdo             [email protected]
3fsyut7             [email protected]
wiuk7k4             [email protected]
ehpjj76                 [email protected]
qw2elyk                 [email protected]
n35h3ms         [email protected]
p3ujrfd             [email protected]
yoyydto                 [email protected]
rp5rvar                 [email protected]
sqh3ckt                     [email protected]
omwksxg                         [email protected]
upkqxh2             [email protected]
tnk52b6             [email protected]
che7ezb         [email protected]
6pqhjod             [email protected]
mq5csuf             [email protected]
afgghmq             [email protected]
jtrpaef                 [email protected]
hnw45un         [email protected]
jftho3y             [email protected]
gsyujwx         [email protected]

==> 1 installed package

Do we have any standardised way of testing the installed gromacs?

giordano · 2024-02-21T12:59:27Z

For future record: as discussed in #49 (comment) we decided to use [email protected] as base compiler, but it turned out that [email protected] is incompatible with this version of GCC (nvcc throws an error if you attempt to use that as base compiler), which for example is a problem for CP2K (spec I'm currently experimenting with: '[email protected] +cuda +mpi +openmp ~sirius cuda_arch=80', based on https://github.com/UCL-RITS/rcps-buildscripts/blob/db896111408d215aec55e820cb250bc576387b21/cp2k-7.1_install).

We will try to use [email protected] as compiler for CUDA packages, and [email protected] for the rest and see how that goes.

Update (2024-02-28): we ran into too many problems with nvhpc (e.g. spack/spack#42879 and spack/spack#32164). Also, nvhpc can't be used to compile all packages, there are many known incompatibilities (e.g. python%nvhpc won't even concretize due to known conflicts). Next step is try to compile most of our stack with [email protected], and then try to use [email protected] for the CUDA packages (hoping we don't run again into the ICEs in the vectoriser mentioned at #49 (comment)).

Update (2024-03-04): [email protected] has fixed the ICEs we ran into with [email protected], so we should be good with using it as the only compiler for the entire stack.

heatherkellyucl · 2024-06-19T16:19:49Z

LAMMPS (GNU, CPU) is going to look like this, as they specify ALL the packages individually where we'd been using presets/most.cmake for the recent releases.

+mpi +openmp are on by default, but I figured since we were specifying everything else it would be worth saying so.

[email protected] +mpi +python +amoeba +asphere +bocs +body +bpm +brownian +cg-dna +cg-spica +class2 +colloid +colvars +compress +coreshell +dielectric +diffraction +dipole +dpd-basic +dpd-meso +dpd-react +dpd-smooth +drude +eff +electrode +extra-compute +extra-dump +extra-fix +extra-molecule +extra-pair +fep +granular +interlayer +kspace +lepton +machdyn +manybody +mc +meam +mesont +misc +ml-iap +ml-pod +ml-snap +mofff +molecule +openmp +opt +orient +peri +phonon +plugin +poems +qeq +reaction +reaxff +replica +rigid +shock +sph +spin +srd +tally +uef +voronoi +yaff

Only thing missing that we had mentioned in the last install was quip, which isn't an option.

For CUDA, build with GNU. Same packages. +cuda cuda_arch=80. Needed doing on GPU node last time.

For Intel, we do one version with +user-intel and one without.

heatherkellyucl · 2024-06-19T17:31:38Z

CASTEP, ORCA, NAMD are all manual-download packages.

You can set a mirror or source_cache to be checked and they should be found in it, or you can specify exactly the location of each package (can be in environment's .yaml) like

https://spack.readthedocs.io/en/latest/packages_yaml.html#assigning-package-attributes

You can assign class-level attributes in the configuration.
packages:
 mpileaks:
   package_attributes:
     # Override existing attributes
     url: http://www.somewhereelse.com/mpileaks-1.0.tar.gz
     # ... or add new ones
     x: 1
Attributes set this way will be accessible to any method executed in the package.py file (e.g. the install() method). Values for these attributes may be any value parseable by yaml.

We have the pkg-store so setting that as a source mirror should work.

heatherkellyucl · 2024-06-20T15:12:08Z

ORCA

[email protected] is still the newest (need to get the package.py from develop for 0.22 for 5.0.4 checksum).

Will build openmpi 4.1.2.

No hurry since we already have this version of ORCA - will add in after other MPI stuff. I think we should be fine using the shared library builds which are the ones Spack has checksums for and are much smaller. The static builds come in three large tar files which would need alterations to the recipe. It has to have an external openmpi in either case.

heatherkellyucl · 2024-06-20T15:54:17Z

Got package.pys in our repos. NAMD and ORCA are just for new versions from spack develop. For CASTEP we want to build 23.1.1 not 21.11 so that needs testing first.

 repos/dev/packages/castep/package.py 
 repos/ucl/packages/namd/package.py 
 repos/ucl/packages/orca/package.py

heatherkellyucl · 2024-06-20T16:06:03Z

Latest GROMACS available in develop:

[email protected] +double
[email protected] +cuda cuda_arch=80 
gromacs@2023 +double +plumed
gromacs@2023 +cuda cuda_arch=80 +plumed

Added from develop:

repos/ucl/packages/gromacs

heatherkellyucl · 2024-06-21T08:28:16Z

See #59 for CASTEP.

heatherkellyucl · 2024-06-21T10:54:26Z

Added manylinux binary cache and our pkgstore to spacksites/settings/initial_site_mirrors.yaml

Might be an issue if it tries reusing Perl from the manylinux cache as there's a missing/circular libcrypt dependency - need to see if that happens.

(To add to an already-existing site, edit /shared/ucl/apps/spack/0.22/hk-initial-stack/spack/etc/spack/mirrors.yaml for example).

heatherkellyucl mentioned this issue Jul 26, 2023

Benchmarking OpenMPI with OSU micro-benchmarks #45

Open

heatherkellyucl mentioned this issue Nov 13, 2023

Initial software: Myriad packages #49

Open

heatherkellyucl mentioned this issue Jun 20, 2024

Building parts of Spack environments on different nodes: option1 (include other concretised environments) #58

Open

heatherkellyucl mentioned this issue Jun 26, 2024

Install Request: LAMMPS with Python in version of 7 Feb 2024 or later UCL-RITS/rcps-buildscripts#575

Open

This was referenced Aug 23, 2024

Install Request: ORCA 6.0 UCL-RITS/rcps-buildscripts#584

Open

Install Request: CP2K 2024.x GPU UCL-RITS/rcps-buildscripts#560

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial software set #44

Initial software set #44

heatherkellyucl commented Jul 19, 2023 •

edited

Loading

heatherkellyucl commented Jul 19, 2023 •

edited

Loading

heatherkellyucl commented Jul 19, 2023

heatherkellyucl commented Jul 25, 2023 •

edited

Loading

krishnakumarg1984 commented Aug 25, 2023 •

edited

Loading

heatherkellyucl commented Aug 29, 2023 •

edited

Loading

krishnakumarg1984 commented Aug 29, 2023

heatherkellyucl commented Aug 29, 2023

heatherkellyucl commented Sep 1, 2023 •

edited

Loading

heatherkellyucl commented Sep 1, 2023

krishnakumarg1984 commented Sep 12, 2023 •

edited

Loading

giordano commented Feb 21, 2024 •

edited

Loading

heatherkellyucl commented Jun 19, 2024

heatherkellyucl commented Jun 19, 2024

heatherkellyucl commented Jun 20, 2024

heatherkellyucl commented Jun 20, 2024

heatherkellyucl commented Jun 20, 2024 •

edited

Loading

heatherkellyucl commented Jun 21, 2024

heatherkellyucl commented Jun 21, 2024 •

edited

Loading

Initial software set #44

Initial software set #44

Comments

heatherkellyucl commented Jul 19, 2023 • edited Loading

Large usage MD codes

Myriad-flavoured packages

Second set

Out of scope for initial set

GPU builds

heatherkellyucl commented Jul 19, 2023 • edited Loading

heatherkellyucl commented Jul 19, 2023

heatherkellyucl commented Jul 25, 2023 • edited Loading

krishnakumarg1984 commented Aug 25, 2023 • edited Loading

heatherkellyucl commented Aug 29, 2023 • edited Loading

krishnakumarg1984 commented Aug 29, 2023

heatherkellyucl commented Aug 29, 2023

heatherkellyucl commented Sep 1, 2023 • edited Loading

heatherkellyucl commented Sep 1, 2023

krishnakumarg1984 commented Sep 12, 2023 • edited Loading

giordano commented Feb 21, 2024 • edited Loading

heatherkellyucl commented Jun 19, 2024

heatherkellyucl commented Jun 19, 2024

heatherkellyucl commented Jun 20, 2024

heatherkellyucl commented Jun 20, 2024

heatherkellyucl commented Jun 20, 2024 • edited Loading

heatherkellyucl commented Jun 21, 2024

heatherkellyucl commented Jun 21, 2024 • edited Loading

heatherkellyucl commented Jul 19, 2023 •

edited

Loading

heatherkellyucl commented Jul 19, 2023 •

edited

Loading

heatherkellyucl commented Jul 25, 2023 •

edited

Loading

krishnakumarg1984 commented Aug 25, 2023 •

edited

Loading

heatherkellyucl commented Aug 29, 2023 •

edited

Loading

heatherkellyucl commented Sep 1, 2023 •

edited

Loading

krishnakumarg1984 commented Sep 12, 2023 •

edited

Loading

giordano commented Feb 21, 2024 •

edited

Loading

heatherkellyucl commented Jun 20, 2024 •

edited

Loading

heatherkellyucl commented Jun 21, 2024 •

edited

Loading