Initial software: Myriad packages

[heatherkellyucl]

From #44

Samtools, BCFTools, HTSlib: UCL-RITS/rcps-buildscripts#532

Samtools, Bwa, Bedtools, Vcftools, picard, GATK (requested by Biosciences) https://ucldata.atlassian.net/browse/AHS-139

Alphafold (would be nice to not have to tell people to go and track down the working containers): UCL-RITS/rcps-buildscripts#529 UCL-RITS/rcps-buildscripts#463

HDF5
Netcdf
BEAST 2.7: UCL-RITS/rcps-buildscripts#498

[heatherkellyucl]

On Myriad, continuing from existing site with compilers only installed:

source spacksites/myriad-utilities/init-spacksites-on-myriad.sh
eval $(spacksites/spacksites spack-setup-env hk-initial-stack)

Looking at adding these to the environment

[email protected] %[email protected]
[email protected] %[email protected]
[email protected] %[email protected]

(want them all to be 1.16 or we get two htslibs, since samtools 1.16.1 wants htslib 1.16).

[heatherkellyucl]

Fun times, ran into spack/spack#35197 with gzip as an htslib dependency.

Works if you get it to use gzip 1.13 instead with spack checksum --add-to-package gzip (which now should be the one in use as everything less has a vulnerability and needs to be deprecated - To Be Pull Requested).

[heatherkellyucl]

gzip is updated in develop branch (https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/gzip/package.py). Need to work out what exactly we do to make sure we include that version here when creating a new environment. Add to local specs?

[giordano]

Since gzip 1.12- is deprecated, the resolver should basically only consider [email protected] (unless you explicitly use --deprecated. If you want to be super explicit, in your config you could have

packages:
  gzip:
    require: "@1.13:"

[heatherkellyucl]

It is only deprecated in the develop branch though - so we won't see that in 0.20 unless we do something to make sure we get that version of the package.py.

[heatherkellyucl]

(We'll get https://github.com/spack/spack/blob/releases/v0.20/var/spack/repos/builtin/packages/gzip/package.py)

[giordano]

So you wouldn't have [email protected] either 😞

[heatherkellyucl]

Yeah. We will need to solve this in the more general case for anything else where we need to alter the package.py for whatever reason. Which is why I was thinking of a local package location too as part of this repo, where new packages and altered packages go.

[heatherkellyucl]

repos.yaml: https://spack.readthedocs.io/en/latest/repositories.html

[heatherkellyucl]

It'll look through the repos in order, so we put ours first and it'll get gzip from there.

On updating the version of Spack we are pinned to, we would need to check whether we should remove packages from our own repo to return to using Spack's ones.

[heatherkellyucl]

[email protected] depends on python and at least a few packages :sadface:

Didn't want to mess with python yet, might have to.

Actually, this might be fine, we just wouldn't have created our view and associated module yet.

[heatherkellyucl]

James started a local repo as repos/dev in the top level in here. We probably want a repos/ucl too for when things are non-dev.

[heatherkellyucl]

Got these after installing all three:

-- linux-rhel7-skylake_avx512 / [email protected] ----------------------
[email protected]                       [email protected]        [email protected]
[email protected]                 [email protected]          [email protected]
[email protected]                         [email protected]           [email protected]
ca-certificates-mozilla@2023-01-10  [email protected]          [email protected]
[email protected]                        [email protected]       [email protected]
[email protected]                          [email protected]        [email protected]
[email protected]                       [email protected]           [email protected]
[email protected]                         [email protected]              [email protected]
[email protected]                     [email protected]         [email protected]
[email protected]                           [email protected]       [email protected]
[email protected]                      [email protected]  [email protected]
[email protected]                         [email protected]    [email protected]
[email protected]                           [email protected]         [email protected]
[email protected]                         [email protected]      [email protected]
[email protected]                       [email protected]     [email protected]
[email protected]                     [email protected]    [email protected]
[email protected]                        [email protected]   [email protected]
[email protected]                       [email protected]   [email protected]
[email protected]                 [email protected]   [email protected]
[email protected]                         [email protected]       [email protected]
[email protected]                       [email protected]     [email protected]
[email protected]                    [email protected]       [email protected]

[heatherkellyucl]

We now have these in this repo:
https://github.com/UCL-ARC/hpc-spack/blob/0.20/repos/ucl/repo.yaml
https://github.com/UCL-ARC/hpc-spack/blob/0.20/repos/ucl/packages/gzip/package.py

[heatherkellyucl]

Added to spacksites/settings/initial_site_repos.yaml

Right now, in my site created without that addition:

spack repo list
==> 2 package repositories.
ucl.arc.hpc.dev    /lustre/scratch/scratch/ccspapp/spack/0.20/hpc-spack/repos/dev
builtin            /lustre/shared/ucl/apps/spack/0.20/hk-initial-stack/spack/var/spack/repos/builtin

[heatherkellyucl]

[email protected] %[email protected] - no deps, goes fine.

vcftools: already have all the dependencies but need to update the versions available, get from develop
[email protected] %[email protected]
Needs some checking as the one from develop depends on zlib-api.

[email protected] %[email protected] - already got all the deps.

[heatherkellyucl]

[email protected] %[email protected] - one dep, openjdk11

[email protected] %[email protected] - from develop, one dep, openjdk17

[heatherkellyucl]

Looked at [email protected] %[email protected] again - for anything in develop that depends on zlib-api, we need that to depend on zlib instead for 0.20. Updated our repos/ucl/packages/vcftools/package.py to do that and added a comment as to why.

[heatherkellyucl]

HDF5: we currently have a serial version with fortran and cxx enabled and an mpi version with fortran enabled. (We have an old serial+threadsafe install, but I think we can ignore this for now unless something else brings it in as a dependency).

[email protected] +cxx +fortran -mpi - got all the deps

[email protected] +fortran +mpi - waiting on the MPI passthrough.

[heatherkellyucl]

NetCDF: we have netcdf with HDF5 (Spack calls it netcdf-c), netcdf-fortran, netcdf-c++. No mpi.

[heatherkellyucl]

Notes from end of yesterday: had some fun with hdf5 versions. (By default it will try to build with MPI on, even for a netcdf that specifies no mpi).

I had tried installing these two and ended up with slightly different concretisations and so two different hashes:

[email protected] +cxx +fortran -mpi %[email protected]
[email protected] -mpi %[email protected] ^[email protected] +cxx +fortran -mpi %[email protected]

The ^[email protected] +cxx +fortran -mpi %[email protected] part specifies the dependency specs.

They were

==> hdf5: Successfully installed hdf5-1.14.1-2-523xubmhtypec6qbccenfcd6yrjkai56
==> Installing hdf5-1.14.1-2-n2ejgnpxlkqidimamznciimgs2s22z3c for netcdf

This lets you see the difference:

spack find -v hdf5
-- linux-rhel7-skylake_avx512 / [email protected] ----------------------
[email protected]+cxx+fortran~hl~ipo~java~map~mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make
[email protected]+cxx+fortran+hl~ipo~java~map~mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make

It was ~hl and +hl.

  hl [off]                --                                on, off                 Enable the high-level library

Let it go with the variants that netcdf uses. Uninstall the hash I don't want:

spack uninstall /523xubmhtypec6qbccenfcd6yrjkai56

[heatherkellyucl]

I think we can leave out netcdf-cxx/netcdf-cxx4 for now, as REPAST is the only thing we currently have that depends on it (netcdf-cxx, the legacy version).

[heatherkellyucl]

Current Spack has BEAST2 2.6.7, need to get 2.7.4 from develop.

[heatherkellyucl]

For this environment, I have changed my /shared/ucl/apps/spack/0.20/hk-initial-stack/spack/var/spack/environments/compilers/spack.yaml to add

  concretizer:
    unify: when_possible
  view: False

This is because it is fine to have Java 11 and Java 17 installed if necessary, or different variants of a package when those are needed, and we do not need a view for all these packages together - it is not intended that someone automatically have them all in their environment at once. (A view requires that everything can be uniquely symlinked into the view directory).

[heatherkellyucl]

Here's the list of everything I added to this environment (which needs recreating):

[email protected] %[email protected]
[email protected] %[email protected]
[email protected]%[email protected]+cxx+fortran~mpi
[email protected] %[email protected] 
[email protected] %[email protected]
[email protected] %[email protected] 
[email protected] %[email protected]
[email protected] %[email protected]
[email protected] %[email protected] 
[email protected] -mpi %[email protected] ^[email protected] +cxx +fortran -mpi %[email protected]
[email protected] %[email protected] ^[email protected] +cxx +fortran -mpi %[email protected]
[email protected] %[email protected]

Anything not available in spack 0.20 comes from repos/ucl/packages/

The only additional package needed is a version of alphafold.

As per comment directly above, it needs this to allow two versions of Java to be installed.

  concretizer:
    unify: when_possible
  view: False

• We should redo the buildcache on Myriad, as these files were reported damaged:

/lustre/shared/ucl/apps/spack/0.20/buildcache/build_cache/linux-rhel7-skylake_avx512-gcc-11.2.1-berkeley-db-18.1.40-diffrysevtavcyynmwqh3juzlophaclr.spec.json.sig
/lustre/shared/ucl/apps/spack/0.20/buildcache/build_cache/linux-rhel7-skylake_avx512-gcc-11.2.1-gcc-12.2.0-pzbagboey5iumrz237ldmjmvtdsaazrb.spec.json.sig
/lustre/shared/ucl/apps/spack/0.20/buildcache/build_cache/linux-rhel7-skylake_avx512-gcc-11.2.1-texinfo-6.5-vk4bsesfczjjrhl7d3rmm4huwjsezaxn.spec.json.sig
/lustre/shared/ucl/apps/spack/0.20/buildcache/build_cache/linux-rhel7-skylake_avx512/gcc-11.2.1/bzip2-1.0.8/linux-rhel7-skylake_avx512-gcc-11.2.1-bzip2-1.0.8-iwbmigw4lpwnlkezmhgqczva6tf2hfxx.spack
/lustre/shared/ucl/apps/spack/0.20/buildcache/build_cache/linux-rhel7-skylake_avx512/gcc-11.2.1/gcc-12.2.0/linux-rhel7-skylake_avx512-gcc-11.2.1-gcc-12.2.0-pzbagboey5iumrz237ldmjmvtdsaazrb.spack
/lustre/shared/ucl/apps/spack/0.20/buildcache/build_cache/linux-rhel7-skylake_avx512/gcc-11.2.1/mpfr-4.1.0/linux-rhel7-skylake_avx512-gcc-11.2.1-mpfr-4.1.0-ilbyeu4bmajiybjcrakptc2ptvz3yfss.spack

@giordano will need to either create the concretised environment in his own space and we'll install the resulting .yaml as ccspapp, or we can give him access to ccspapp and he can do the build in /shared/ucl/apps/spack/0.20/ and when complete can write it to the cache himself. May be better to start off in own space.

• We are getting a build node set aside on Myriad - once we have that, need to make sure he has access.

We have not looked at the module files created yet - all of the above should have modules created for them as they are requested explicitly. Will need to check on LIBRARY_PATH, LD_LIBRARY_PATH for the libraries as they are likely to be wanted in external compilations.

Need to check if anything else needs to be a top-level spec with a module generated. Like Java.

[giordano]

A quick summary of my work so far:

• it's better to specify a version of cuda compatible with the Nvidia drivers installed on the system, @heatherkellyucl recommended using [email protected]. We need to add a copy of cuda/package.py to our repo to add the newer version
• most packages could compile smoothly, but I've been struggling a lot with the py-alphafold stack:
  • [email protected]+cuda requires the patch from Switches to using FindCUDAToolkit openmm/openmm#3154 for building on a node without a GPU. This was fixed in Spack with openmm: Apply patch use FindCUDAToolkit spack/spack#42437. We'll need to add a copy of openmm/package.py to our repo to apply the patch
  • openmm from v7.5.1 to v7.7.0 (I haven't tried different versions) cannot be compiled with GCC 12.2.0 because the compiler crashes with an ICE (Internal Compiler Error). We can't do anything about it, apart from using a different compiler. I can confirm [email protected]+cuda could be compiled correctly with %[email protected]
  • there's a bug in the py-alphafold recipe in Spack, there's a reference to a non-permanent URL on GitHub which of course now doesn't work anymore. I opened a PR to fix this at py-alphafold: use permanent link for openmm patch spack/spack#42461. Also in this case we'll need a copy of py-alphafold/package.py which includes this fix
  • I'm now getting a few errors during compilation of [email protected] (another dependency of py-alphafold....), I'll look into them next week.

This is a kind of provisional environment I got so far, but do not that I haven't attempted to rebuild all of it from scratch, in particular I haven't rebuilt anything with %[email protected] apart from the py-alphafold stack (but all other packages were successful with %[email protected], this may be useful information in case we want to ditch py-alphafold from this round)

spack:
  specs:
  - matrix:
    - - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected] ^[email protected]'
      - '[email protected] +cxx +fortran -mpi +hl'
      - '[email protected] +fortran +mpi +hl'
      - '[email protected] -mpi'
      - '[email protected]'
      - '[email protected]'
      - '[email protected]'
    - - '%[email protected]'
  concretizer:
    unify: when_possible
  view: false
  modules:
    prefix_inspections:
      lib: ["LD_LIBRARY_PATH", "LIBRARY_PATH"]
      lib64: ["LD_LIBRARY_PATH", "LIBRARY_PATH"]

[giordano]

Following discussion on Slack with @heatherkellyucl, I removed py-alphafold from the initial stack, switched to using [email protected] as first compiler (see #52), and this is the environment I'm currently using:

spack:
  specs:
  - matrix:
    - - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected]'
      - '[email protected] +cxx +fortran -mpi +hl'
      - '[email protected] +fortran +mpi +hl'
      - '[email protected]'
      - '[email protected] -mpi'
      - '[email protected]'
      - '[email protected]'
      # - '[email protected]'
      - '[email protected]'
    - - '%[email protected]'
  concretizer:
    unify: when_possible
  view: false
  modules:
    default:
      tcl:
        projections:
          all:  '{name}/{version}/{compiler.name}-{compiler.version}'
          ^mpi: '{name}/{version}-{^mpi.name}/{compiler.name}-{compiler.version}'
    prefix_inspections:
      lib: ["LD_LIBRARY_PATH", "LIBRARY_PATH"]
      lib64: ["LD_LIBRARY_PATH", "LIBRARY_PATH"]

The modules section

• sets the name of the modules with the projections setting. I tried to follow the same pattern I saw on Myriad, of appending "mpi" to the version number, when a package is built with MPI support, but this can be tweaked if necessary
• also sets LD_LIBRARY_PATH/LIBRARY_PATH as requested above

For example:

$ module show hdf5/1.14.1-2/gcc-13.1.0 
-------------------------------------------------------------------
./hdf5/1.14.1-2/gcc-13.1.0:

module-whatis    HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data.  
module           load pkgconf/1.9.5/gcc-13.1.0 
module           load zlib/1.2.13/gcc-13.1.0 
prepend-path     --delim : LD_LIBRARY_PATH /lustre/scratch/scratch/cceamgi/repo/hpc-spack/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p/linux-rhel7-skylake_avx512/gcc-13.1.0/hdf5-1.14.1-2-o2pm6juc4ewwe5wgj56ukyipf5w4ccuf/lib 
prepend-path     --delim : LIBRARY_PATH /lustre/scratch/scratch/cceamgi/repo/hpc-spack/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p/linux-rhel7-skylake_avx512/gcc-13.1.0/hdf5-1.14.1-2-o2pm6juc4ewwe5wgj56ukyipf5w4ccuf/lib 
prepend-path     --delim : PATH /lustre/scratch/scratch/cceamgi/repo/hpc-spack/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p/linux-rhel7-skylake_avx512/gcc-13.1.0/hdf5-1.14.1-2-o2pm6juc4ewwe5wgj56ukyipf5w4ccuf/bin 
prepend-path     --delim : PKG_CONFIG_PATH /lustre/scratch/scratch/cceamgi/repo/hpc-spack/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p/linux-rhel7-skylake_avx512/gcc-13.1.0/hdf5-1.14.1-2-o2pm6juc4ewwe5wgj56ukyipf5w4ccuf/lib/pkgconfig 
prepend-path     --delim : CMAKE_PREFIX_PATH /lustre/scratch/scratch/cceamgi/repo/hpc-spack/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p/linux-rhel7-skylake_avx512/gcc-13.1.0/hdf5-1.14.1-2-o2pm6juc4ewwe5wgj56ukyipf5w4ccuf/. 
-------------------------------------------------------------------

$ module show hdf5/1.14.1-2-mpi/gcc-13.1.0 
-------------------------------------------------------------------
./hdf5/1.14.1-2-mpi/gcc-13.1.0:

module-whatis    HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data.  
module           load openmpi/4.1.5/gcc-13.1.0 
module           load pkgconf/1.9.5/gcc-13.1.0 
module           load zlib/1.2.13/gcc-13.1.0 
prepend-path     --delim : LD_LIBRARY_PATH /lustre/scratch/scratch/cceamgi/repo/hpc-spack/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p/linux-rhel7-skylake_avx512/gcc-13.1.0/hdf5-1.14.1-2-fsznlyxgdczelstzmozmthdew4rffdr7/lib 
prepend-path     --delim : LIBRARY_PATH /lustre/scratch/scratch/cceamgi/repo/hpc-spack/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p/linux-rhel7-skylake_avx512/gcc-13.1.0/hdf5-1.14.1-2-fsznlyxgdczelstzmozmthdew4rffdr7/lib 
prepend-path     --delim : PATH /lustre/scratch/scratch/cceamgi/repo/hpc-spack/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p/linux-rhel7-skylake_avx512/gcc-13.1.0/hdf5-1.14.1-2-fsznlyxgdczelstzmozmthdew4rffdr7/bin 
prepend-path     --delim : PKG_CONFIG_PATH /lustre/scratch/scratch/cceamgi/repo/hpc-spack/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p/linux-rhel7-skylake_avx512/gcc-13.1.0/hdf5-1.14.1-2-fsznlyxgdczelstzmozmthdew4rffdr7/lib/pkgconfig 
prepend-path     --delim : CMAKE_PREFIX_PATH /lustre/scratch/scratch/cceamgi/repo/hpc-spack/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p/linux-rhel7-skylake_avx512/gcc-13.1.0/hdf5-1.14.1-2-fsznlyxgdczelstzmozmthdew4rffdr7/. 
-------------------------------------------------------------------

[ccaefch0523]

On Myriad as ccspapp in my spacksite fc-myriad-stack, I uninstalled [email protected] with gcc12.2.0 then installed the same OpenMPI version 4.1.5 with gcc13.1.0 and passed the following fabrics list:

 spack install [email protected]%[email protected] fabrics=psm2,cma,ucx,knem,xpmem,ofi  schedulers=sge
...
==> openmpi: Successfully installed openmpi-4.1.5-vpih2j6ihyyej4z4amggnvzf5uyzgtcj
  Stage: 10.71s.  Autoreconf: 0.01s.  Configure: 5m 57.87s.  Build: 10m 39.48s.  Install: 1m 17.02s.  Post-install: 1.38s.  Total: 18m 7.92s
[+] /lustre/shared/ucl/apps/spack/0.20/fc-myriad-stack/spack/opt/spack/[padded-to-255-chars]/linux-rhel7-skylake_avx512/gcc-13.1.0/openmpi-4.1.5-vpih2j6ihyyej4z4amggnvzf5uyzgtcj

I also installed the NVDIA package nvhpc

 spack install nvhpc
==> Installing nvhpc-23.3-szho75x5rpschjd4e7yopuawdfg3bthh
==> No binary for nvhpc-23.3-szho75x5rpschjd4e7yopuawdfg3bthh found: installing from source
==> Fetching https://developer.download.nvidia.com/hpc-sdk/23.3/nvhpc_2023_233_Linux_x86_64_cuda_multi.tar.gz
%nvhpc==> No patches needed for nvhpc
==> nvhpc: Executing phase: 'install'
==> nvhpc: Successfully installed nvhpc-23.3-szho75x5rpschjd4e7yopuawdfg3bthh
  Stage: 14m 1.52s.  Install: 8m 28.00s.  Post-install: 3m 21.21s.  Total: 25m 50.78s

I will then try installing Gromacs + cuda using nvhpc compiler