Skip to content

Shannon

Glen Hansen edited this page Feb 11, 2019 · 9 revisions

Detailed build and run instructions with Serial, OpenMP and CUDA node (10/13/2016)

Please refer to SAND report written by Jerry Watkins, which can be found here.

These are the build instructions for Shannon (11/18/2015)

Please email [email protected] with comments or questions.

Shannon is integrated by Appro

  • Current configuration installed 9/2014
  • 22 compute nodes
  • Each node has two 8 core Sandy Bridge Xeon E5-2670 processors @ 2.6 GHz
  • Each node has 128 GB - DDR3 memory 1600 MHz
  • 6 nodes have 2x NVIDIA K20x (Kepler), 12 have 4x K20x, 4 have 4x K40m, and 9 have 2x K80s
  • 2688 732MHz cores
  • Mellanox QDR Infiniband

Set up your build environment

Here is the module load sequence used

	module purge
	module load openmpi/1.10.1/gnu/4.7.2/cuda/7.5.7
	export CUDA_LAUNCH_BLOCKING=1

Install the required Third Party Libraries (TPLs) for Trilinos

These install directions assume that a base install directory of /home/gahanse/gcc-4.7.2/ompi-1.10.1 is used

Note that "shannon" has prebuilt versions of zlib, hfd5, and netcdf. These are actually used, but we show build instructions on how one would build these libraries if they do not exist. We mark these as (optional).

Building ZLib (optional)

  • Version 1.2.8

      cd /home/gahanse/src
      wget http://zlib.net/zlib-1.2.8.tar.gz
      tar -xvf zlib-1.2.8.tar.gz
      cd zlib-1.2.8
      CC=mpicc CFLAGS="-O3" ./configure --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1
      make install
    

Building HWLOC (optional)

  • Version 1.9

      ./configure --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1 CC=mpicc CXX=mpicxx
    

Building HDF5 (optional)

  • Version 1.8.13

      cd /home/gahanse/src/
      wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.13.tar.bz2
      tar -xvf hdf5-1.8.13.tar.gz
      cd hdf5-1.8.13
      ./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" \
      --enable-parallel --with-zlib=/home/gahanse/gcc-4.7.2/ompi-1.10.1 --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1
      make -j 8
      make install
    

Building Netcdf (optional)

  • Version 4.3.3.1

      cd /home/gahanse/src/
      wget ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.3.3.1.tar.gz
      tar -xvf netcdf-4.3.3.1.tar.gz
      cd netcdf-4.3.3.1
    
  • Edit the file include/netcdf.h. Make the following edits near line 228:

      #define NC_MAX_DIMS    65536    /* max dimensions per file */
      #define NC_MAX_ATTRS    8192    
      #define NC_MAX_VARS   524288    /* max variables per file */
      #define NC_MAX_NAME   256 
      #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */
    
  • Now, configure build:

      ./configure --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1 CC=mpicc FC=mpifort CXX=mpicxx \
      CFLAGS="-I/home/gahanse/gcc-4.7.2/ompi-1.10.1/include -fpic -O3" \
      CXXFLAGS="-I/home/gahanse/gcc-4.7.2/ompi-1.10.1/include -fPIC -O3" \
      LDFLAGS=-L/home/gahanse/gcc-4.7.2/ompi-1.10.1/lib \
      FCFLAGS="-fPIC -O3" --disable-fsync --disable-cdmremote --disable-dap --disable-doxygen --enable-netcdf-4
    
      make -j 8 
      make install
    

Building boost

  • Version 1.59.0

      cd /home/gahanse/src/
      wget http://sourceforge.net/projects/boost/files/boost/1.59.0/boost_1_59_0.tar.gz/download
      tar -xvf boost_1_59_0.tar.gz
      cd boost_1_59_0
    
  • Build boost with the commands

      echo "using gcc :  4.7.2 : /home/projects/gcc/4.7.2/bin/g++ ;" > ~/user-config.jam
      echo "using mpi : /home/projects/x86-64-sandybridge-nvidia/openmpi/1.10.1/gnu/4.7.2/cuda/7.5.7/bin/mpicxx ;" >> ~/user-config.jam
      ./bootstrap.sh \
      --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception \
      --prefix=/home/gahanse/gcc-4.7.2/ompi-1.10.1
    
      ./b2 -j 8
      ./b2 -j 8 install
    

Building ParMetis

  • Version 4.0.3

      cd /home/gahanse/src/
      wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
      tar -xvf parmetis-4.0.3.tar.gz
      cd parmetis-4.0.3
    
  • Edit metis.h to use an IDXTYPEWIDTH of 64

      cd metis/include
      vi metis.h
      #define IDXTYPEWIDTH 64
      #define REALTYPEWIDTH 64
    
  • Run CMake to build parmetis

      cd /home/gahanse/src/parmetis-4.0.3
      cd build
      export metis=/home/gahanse/src/parmetis-4.0.3/metis
    
      cmake \
      -DCMAKE_INSTALL_PREFIX=/home/gahanse/gcc-4.7.2/ompi-1.10.1 \
      -DMETIS_PATH=$metis \
      -DGKLIB_PATH=$metis/GKlib \
      -DCMAKE_C_COMPILER=mpicc \
      -DCMAKE_CXX_COMPILER=mpicxx \
      -DMPI_INCLUDE_PATH=/home/gahanse/gcc-4.7.2/ompi-1.10.1/include \
      -DCMAKE_C_FLAGS="-fPIC -O3" \
      -DCMAKE_CXX_FLAGS="-fPIC -O3" \
      ..
    
      make -j 8
      make install
    
      cp libmetis/libmetis.a /home/gahanse/gcc-4.7.2/ompi-1.10.1/lib
      cp $metis/include/metis.h /home/gahanse/gcc-4.7.2/ompi-1.10.1/include
    

Build Trilinos

  • Download Trilinos

      cd /home/gahanse/src
      git clone https://github.com/trilinos/Trilinos.git
    
  • Download SCOREC

      cd /home/gahanse/src/Trilinos
      git clone [email protected]:SCOREC/core.git SCOREC
    
  • Configure the Trilinos build

      mkdir build
      cd build
      ./do-configure
    
  • Example do-configure script executed above

      #!/bin/sh
      #
    
      export TRILINOS_HOME=/home/gahanse/Codes/Trilinos
      BUILD_DIR=`pwd`
      INSTALL_DIR=/home/gahanse/trilinos/MPI_NGCC
      BOOST_DIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4
      MPI_BASE_DIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4
      NETCDF=/home/gahanse/gcc-4.9.0/mpich-3.1.4
      HDFDIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4
      PARMETISDIR=/home/gahanse/gcc-4.9.0/mpich-3.1.4
      MKL_PATH=/opt/intel/mkl/lib/intel64
      HWLOC_PATH=/home/gahanse/gcc-4.9.0/mpich-3.1.4
      LABLAS_LIBRARIES="-L$MKL_PATH/lib/intel64 -Wl,--start-group $MKL_PATH/mkl/lib/intel64/libmkl_intel_lp64.a $MKL_PATH/mkl/lib/intel64/
      
      cmake \
          -Wno-dev \
          -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
          -D Trilinos_ENABLE_SCOREC:BOOL=ON \
          -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
          -D CMAKE_BUILD_TYPE:STRING=NONE \
          -D CMAKE_CXX_COMPILER:FILEPATH="/home/gahanse/Codes/Trilinos/build/nvcc_wrapper_gh" \
          -D CMAKE_C_COMPILER:FILEPATH="mpicc" \
          -D CMAKE_Fortran_COMPILER:FILEPATH="mpifort" \
          -D CMAKE_CXX_FLAGS:STRING="-DNDEBUG" \
          -D Trilinos_CXX11_FLAGS:STRING="-std=c++11 --expt-extended-lambda --expt-relaxed-constexpr -Wno-unused-local-typedefs -Wno-sign-
          -D CMAKE_C_FLAGS:STRING="-O3 -DNDEBUG" \
          -D CMAKE_Fortran_FLAGS:STRING="-O3 -DNDEBUG" \
          -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
          -D Tpetra_INST_INT_LONG_LONG:BOOL=OFF \
          -D Tpetra_INST_INT_INT:BOOL=ON \
          -D Tpetra_INST_DOUBLE:BOOL=ON \
          -D Tpetra_INST_FLOAT:BOOL=OFF \
          -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF \
          -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF \
          -D Tpetra_INST_INT_LONG:BOOL=OFF \
          -D Tpetra_INST_INT_UNSIGNED:BOOL=OFF \
          -D Zoltan_ENABLE_ULONG_IDS:BOOL=OFF \
          -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=ON \
          -D Rythmos_ENABLE_DEBUG:BOOL=ON \
          -D Trilinos_ENABLE_Kokkos:BOOL=ON \
          -D Phalanx_KOKKOS_DEVICE_TYPE:STRING="CUDA" \
          -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" \
          -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF \
          -D Kokkos_ENABLE_Serial:BOOL=ON \
          -D Kokkos_ENABLE_OpenMP:BOOL=OFF \
          -D Kokkos_ENABLE_Pthread:BOOL=OFF \
          -D Kokkos_ENABLE_Cuda:BOOL=ON \
          -D TPL_ENABLE_CUDA:BOOL=ON \
          -D Kokkos_ENABLE_Cuda_UVM:BOOL=ON \
          -D Kokkos_ENABLE_Cuda_LAMBDA:BOOL=ON \
          -D TPL_ENABLE_CUSPARSE:BOOL=ON \
      \
          -D TPL_ENABLE_MPI:BOOL=ON \
          -D MPI_BASE_DIR:PATH=$MPI_BASE_DIR \
      \
          -D TPL_ENABLE_Pthread:BOOL=OFF \
      \
          -D TPL_ENABLE_Boost:BOOL=ON \
          -D TPL_ENABLE_BoostLib:BOOL=ON \
          -D TPL_ENABLE_BoostAlbLib:BOOL=ON \
          -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
          -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
          -D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
          -D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
          -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
          -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
      \
          -D TPL_ENABLE_Netcdf:STRING=ON \
          -D Netcdf_INCLUDE_DIRS:PATH=$NETCDF/include \
          -D Netcdf_LIBRARY_DIRS:PATH=$NETCDF/lib \
      \
          -D TPL_ENABLE_HDF5:STRING=ON \
          -D HDF5_INCLUDE_DIRS:PATH=$HDFDIR/include \
          -D HDF5_LIBRARY_DIRS:PATH=$HDFDIR/lib \
      \
          -D TPL_ENABLE_Zlib:STRING=ON \
          -D Zlib_INCLUDE_DIRS:PATH=$HDFDIR/include \
          -D Zlib_LIBRARY_DIRS:PATH=$HDFDIR/lib \
      \
          -D TPL_ENABLE_BLAS:STRING=ON \
          -D TPL_ENABLE_LAPACK:STRING=ON \
          -D BLAS_LIBRARY_DIRS:FILEPATH=${MKL_PATH} \
          -D BLAS_LIBRARY_NAMES="mkl_intel_lp64;mkl_sequential;mkl_core;m" \
          -D LAPACK_LIBRARY_NAMES="" \
      \
          -D TPL_ENABLE_ParMETIS:STRING=ON \
          -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETISDIR}/include" \
          -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETISDIR}/lib" \
      \
          -D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \
          -D Trilinos_EXTRA_LINK_FLAGS:STRING="-L${HDFDIR}/lib -lnetcdf -lhdf5_hl -lhdf5 -lz" \
      \
          -D Trilinos_ENABLE_Moertel:BOOL=OFF \
          -D Trilinos_ENABLE_TriKota:BOOL=OFF \
          -D Trilinos_ENABLE_SEACASSVDI:BOOL=OFF \
          -D Trilinos_ENABLE_SEACASPLT:BOOL=OFF \
          -D Trilinos_ENABLE_SEACASBlot:BOOL=OFF \
          -D Trilinos_ENABLE_STK:BOOL=ON \
          -D Trilinos_ENABLE_STKUtil:BOOL=ON \
          -D Trilinos_ENABLE_STKTopology:BOOL=ON \
          -D Trilinos_ENABLE_STKMesh:BOOL=ON \
          -D Trilinos_ENABLE_STKIO:BOOL=ON \
          -D Trilinos_ENABLE_STKTransfer:BOOL=ON \
          -D Trilinos_ENABLE_STKUnit_tests:BOOL=OFF \
          -D Trilinos_ENABLE_STKDoc_tests:BOOL=OFF \
          -D TPL_ENABLE_X11:BOOL=OFF \
          -D Trilinos_ENABLE_ThreadPool:BOOL=OFF \
          -D Trilinos_ENABLE_Teko:BOOL=OFF \
          -D Trilinos_ENABLE_MueLu:BOOL=ON \
          -D Trilinos_ENABLE_Amesos2:BOOL=ON \
          -D Amesos2_ENABLE_KLU2:BOOL=ON \
          -D BUILD_SHARED_LIBS:BOOL=OFF \
          -D Trilinos_ENABLE_OptiPack:BOOL=OFF \
          -D Trilinos_ENABLE_GlobiPack:BOOL=OFF \
      \
      $EXTRA_ARGS \
      ${TRILINOS_HOME}
      
      exit 0
    
  • Build Trilinos

      make -j 8
      make install
    

Build Albany

  • Clone the repo

      cd /home/gahanse/src
      git clone [email protected]:SNLComputation/Albany.git
      cd Albany
    
  • Configure the Albany build

      mkdir build
      cd build
      ./do-configure
    
  • CMake configure script ./do-configure:

      #!/bin/sh
      #
      
      export TRILINOS_HOME=/home/gahanse/trilinos/MPI_CUVM
      
      cmake \
        -DALBANY_TRILINOS_DIR:PATH=${TRILINOS_HOME} \
        -DENABLE_LCM:BOOL=ON \
        -DENABLE_LCM_SPECULATIVE:BOOL=OFF \
        -DENABLE_HYDRIDE:BOOL=OFF \
        -DENABLE_SCOREC:BOOL=ON \
        -DENABLE_SG_MP:BOOL=OFF \
        -DENABLE_FELIX:BOOL=ON \
        -DENABLE_AERAS:BOOL=ON \
        -DENABLE_QCAD:BOOL=OFF \
        -DENABLE_MOR:BOOL=OFF \
        -DENABLE_ATO:BOOL=OFF \
        -DENABLE_ASCR:BOOL=OFF \
        -DENABLE_CHECK_FPE:BOOL=OFF \
        -DENABLE_ALBANY_EPETRA_EXE:BOOL=ON \
        -DENABLE_KOKKOS_UNDER_DEVELOPMENT:BOOL=ON \
      \
      ..
      
      exit 0
    
  • Build Albany

      make -j 8
      make install
    
  • Run Albany tests

      salloc -n 4 -N 4 -p stella
      srun ctest
              (we need "srun" in purpose to get tests run on the compute node)
    
  • Run Albany executables salloc -N 1 -p stella srun -n 1 ./Alabny.exe input.xml (we need "srun" in purpose to get tests run on the compute node)

Clone this wiki locally