Westley

Set up your build environment

This build is based on the 2016 Intel compilers

	. /opt/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64
	export PATH=/usr/local/trilinos/MPI_REL/bin:/usr/local/gcc-5.1.0/bin:/opt/intel/bin:/usr/local/bin:/usr/lib64/qt-3.3/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:$HOME/bin
	. /opt/intel/impi/5.1.2.150/bin64/mpivars.sh
	export LD_LIBRARY_PATH=/opt/intel/composer_xe_2015.3.187/compiler/lib/mic:/usr/local/gcc-5.1.0/lib64:$LD_LIBRARY_PATH
	export I_MPI_MIC=1

These install directions assume that a base install directory of /usr/local/mic is used

Install the required Third Party Libraries (TPLs) for Trilinos

Build HWLOC

• HWLOC is an optional package that can be used for thread pinning by Kokkos. If you are not building Kokkos, or if you don't care about thread pinning, please skip.

• Version 1.10

    cd /usr/local/src
    wget http://www.open-mpi.org/software/hwloc/v1.10/downloads/hwloc-1.10.0.tar.bz2
    tar -xvf hwloc-1.10.0.tar.bz2
    cd hwloc-1.10.0
    ./configure CC=mpiicc CFLAGS="-O3 -mmic -fopenmp" CXX=mpiicpc \
    	CXXFLAGS="-O3 -mmic -fopenmp" LDFLAGS=-static \
    	--host=x86_64-k1om-linux --enable-shared=no \
    	--enable-static=yes --prefix=/usr/local/mic
    make -j 8
    make install
  

Building ZLib

• Version 1.2.8

    cd /usr/local/src
    wget http://zlib.net/zlib-1.2.8.tar.gz
    tar -xvf zlib-1.2.8.tar.gz
    cd zlib-1.2.8
    CC=mpiicc CXX=mpiicpc CFLAGS="-O3 -mmic -fopenmp" CXXFLAGS="-O3 -mmic -fopenmp" \
    	./configure --64 --static --prefix=/usr/local/mic
    make install
  

Building HDF5

• Version 1.8.17

Cross compiling HDF5 for the MIC can be a challenge. Please see the instructions at:

	http://linux4hippos.blogspot.com/2013/05/cross-compile-hdf5-for-intel-xeon-phi.html

Build Instructions: Note

• The above website is based on HDF5-1.8.9 but they work with version HDF5-1.8.17

• The above website, step (1), instructs one to set TEST_SCRIPT = in several files. These instructions were followed here.

• as_fn_exit was commented out in three places in the configure script as described in step (2) above

    cd hdf5-1.8.17
    ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc CXXFLAGS="-O3 -mmic -fopenmp" \
    	CFLAGS="-O3 -mmic -fopenmp" FCFLAGS="-O3 -mmic -fopenmp" -host=x86_64-k1om-linux \
    	--enable-parallel --enable-shared=no --with-zlib=/usr/local/mic --prefix=/usr/local/mic
    make -j 8
  

Note that the build fails as expected (above website, step (6)). Follow the instructions given to proceed.

Building parallel-netcdf

• Version 1.7.0

• Configure and build:

    ./configure --prefix=/usr/local/mic MPICC=mpiicc MPIF77=mpiifort MPIF90=mpiifort MPICXX=mpiicpc \
    CXXFLAGS="-O3 -mmic -fopenmp" CFLAGS="-O3 -mmic -fopenmp" \
    F77FLAGS="-O3 -mmic -fopenmp" F90FLAGS="-O3 -mmic -fopenmp" -host=x86_64-k1om-linux
    make -j 8 
    make install
  

Building Netcdf

• Version 4.4.0

• Edit the file include/netcdf.h. Make the following edits near line 265:

    #define NC_MAX_DIMS    65536    /* max dimensions per file */
    #define NC_MAX_ATTRS    8192    
    #define NC_MAX_VARS   524288    /* max variables per file */
    #define NC_MAX_NAME   256 
    #define NC_MAX_VAR_DIMS      8     /* max per variable dimensions */
  

• Now, configure using CMake and build:

    cd hdf5-1.8.17
    ./configure CC=mpiicc FC=mpiifort CXX=mpiicpc CXXFLAGS="-O3 -mmic -fopenmp -I/usr/local/mic/include" \
    	CFLAGS="-O3 -mmic -fopenmp -I/usr/local/mic/include" \
        LDFLAGS="-O3 -mmic -fopenmp -L/usr/local/mic/lib" \
        FCFLAGS="-O3 -mmic -fopenmp -I/usr/local/mic/include" -host=x86_64-k1om-linux \
    	--prefix=/usr/local/mic --enable-netcdf-4 --disable-dap --enable-shared=no \
        --disable-utilities --disable-testsets --enable-pnetcdf
    make -j 8
    make install
  

Building boost

• Version 1.58.0

    cd /usr/local/src/
    wget http://sourceforge.net/projects/boost/files/boost/1.58.0/boost_1_58_0.tar.gz/download boost_1_58_0.tar.gz
    tar -xvf boost_1_58_0.tar.gz
    cd boost_1_58_0
  

• Build boost with the commands

    echo "using intel : mic : : <cxxflags>\"-O3 -mmic -fopenmp\" ;" \
    	>> ~/user-config.jam
    echo "using mpi : /opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin/mpiicpc : <cxxflags>\"-I/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/include -mmic -fopenmp\" ;" \
    	>> ~/user-config.jam
    ./bootstrap.sh --with-toolset=intel \
    --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception \
    --prefix=/usr/local/mic
    ./b2 -j 8
    ./b2 -j 8 install
  

• The above bjam command gave me an error, so I removed the following lines from project-config.jam:

    if ! intel-darwin in [ feature.values <toolset> ]
    {
        using intel-darwin ;
    }
  

Building ParMetis

• Version 4.0.3

    cd /usr/local/src/
    wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz
    tar -xvf parmetis-4.0.3.tar.gz
    cd parmetis-4.0.3
  

• Edit metis.h to use an IDXTYPEWIDTH of 64

    cd metis/include
    vi metis.h
    #define IDXTYPEWIDTH 64
    #define REALTYPEWIDTH 64
  

• Run CMake to build parmetis

    cd /usr/local/src/parmetis-4.0.3
    cd build
    export metis=/usr/local/src/parmetis-4.0.3/metis
  
    cmake \
    -DCMAKE_INSTALL_PREFIX=/usr/local/mic \
    -DMETIS_PATH=$metis \
    -DGKLIB_PATH=$metis/GKlib \
    -DCMAKE_C_COMPILER=mpiicc \
    -DCMAKE_C_FLAGS="-O3 -mmic -fopenmp" \
    -DCMAKE_CXX_COMPILER=mpiicpc \
    -DCMAKE_CXX_FLAGS="-O3 -mmic -fopenmp" \
    ..
  
    make -j 8
    make install
  
    cp libmetis/libmetis.a /usr/local/mic/lib
    cp $metis/include/metis.h /usr/local/mic/include
  

Build Trilinos

• Download Trilinos

    cd /usr/local/src
    git clone [email protected]:trilinos/Trilinos.git
  

• Download SCOREC

    cd /usr/local/src/Trilinos
    git clone [email protected]:SCOREC/core.git SCOREC
  

• Configure the Trilinos build

    mkdir build
    cd build
    ./do-configure
  

• Example do-configure script executed above

    #!/bin/sh
    #
    
    export TRILINOS_HOME=/usr/local/src/Trilinos
    BUILD_DIR=`pwd`
    INSTALL_DIR=/usr/local/mic/trilinos/MPI_MIC
    BOOST_DIR=/usr/local/mic/boost-1.58.0
    MPI_BASE_DIR=/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64
    NETCDF=/usr/local/mic
    HDFDIR=/usr/local/mic
    ZLIB_DIR=/usr/local/mic
    PARMETISDIR=/usr/local/mic
    HWLOC_PATH=/usr/local/mic
    INTEL_DIR=/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/mic
    
    cmake \
        -Wno-dev \
        -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_HOME/sampleScripts/AlbanySettings.cmake \
        -D Trilinos_ENABLE_SCOREC:BOOL=ON \
        -D SCOREC_DISABLE_STRONG_WARNINGS:BOOL=ON \
        -D CMAKE_BUILD_TYPE:STRING=RELEASE \
        -D TPL_FIND_SHARED_LIBS:BOOL=OFF \
        -D BUILD_SHARED_LIBS:BOOL=OFF \
        -D TPL_ENABLE_MPI:BOOL=ON \
        -D MPI_BASE_DIR:PATH="$MPI_BASE_DIR"\
        -D CMAKE_CXX_COMPILER:FILEPATH="${MPI_BASE_DIR}/bin/mpiicpc" \
        -D CMAKE_C_COMPILER:FILEPATH="${MPI_BASE_DIR}/bin/mpiicc" \
        -D CMAKE_Fortran_COMPILER:FILEPATH="${MPI_BASE_DIR}/bin/mpiifort" \
        -D CMAKE_AR:FILEPATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/bin/intel64_mic/xiar \
        -D CMAKE_LINKER:FILEPATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/bin/intel64_mic/xild \
        -D CMAKE_CXX_FLAGS:STRING="-O3 -mmic -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX -DPREC_TIMER -restrict -fasm-blocks -DDEVICE=1wq  -fopenmp" \
        -D CMAKE_C_FLAGS:STRING="-O3 -mmic -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX -DPREC_TIMER -restrict -fasm-blocks -DDEVICE=1wq  -fopenmp" \
        -D CMAKE_Fortran_FLAGS:STRING="-O3 -mmic -DPREC_TIMER -fopenmp" \
        -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
        -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
        -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
        -D Trilinos_ENABLE_Teuchos:BOOL=ON \
       \
        -D Trilinos_WARNINGS_AS_ERRORS_FLAGS:STRING="" \
        -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \
        -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
       \
        -D Trilinos_ENABLE_SEACASNemesis:BOOL=ON \
        -D Trilinos_ENABLE_Zoltan:BOOL=ON \
        -D Zoltan_ENABLE_METIS:BOOL=OFF \
        -D Trilinos_ENABLE_Kokkos:BOOL=ON \
        -D Trilinos_ENABLE_KokkosCore:BOOL=ON \
        -D Trilinos_ENABLE_KokkosContainers:BOOL=ON \
        -D Trilinos_ENABLE_KokkosExample:BOOL=OFF \
        -D Trilinos_EXTRA_LINK_FLAGS:STRING="$NETCDF/lib/libnetcdf.a;$HDFDIR/lib/libhdf5_hl.a;$HDFDIR/lib/libhdf5.a;$ZLIB_DIR/lib/libz.a" \
       \
        -D Kokkos_ENABLE_Serial:BOOL=OFF \
        -D Kokkos_ENABLE_OpenMP:BOOL=ON \
        -D Kokkos_ENABLE_Pthread:BOOL=OFF \
        -D Kokkos_ENABLE_Cuda:BOOL=OFF \
        -D Trilinos_ENABLE_OpenMP:BOOL=ON \
        -D TPL_ENABLE_CUDA:BOOL=OFF \
        -D TPL_ENABLE_CUSPARSE:BOOL=OFF \
    \
        -D TPL_ENABLE_Pthread:BOOL=OFF \
        -D TPL_ENABLE_HWLOC:BOOL=OFF \
    \
        -D TPL_ENABLE_Boost:BOOL=ON \
        -D TPL_ENABLE_BoostLib:BOOL=ON \
        -D TPL_ENABLE_BoostAlbLib:BOOL=ON \
        -D Boost_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
        -D Boost_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
        -D BoostLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
        -D BoostLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
        -D BoostAlbLib_INCLUDE_DIRS:PATH=$BOOST_DIR/include \
        -D BoostAlbLib_LIBRARY_DIRS:PATH=$BOOST_DIR/lib \
    \
        -D TPL_ENABLE_Netcdf:STRING=ON \
        -D TPL_Netcdf_PARALLEL:BOOL=ON \
        -D Netcdf_INCLUDE_DIRS:PATH=$NETCDF/include \
        -D TPL_Netcdf_LIBRARIES:FILEPATH="$NETCDF/lib/libnetcdf.a;$HDFDIR/lib/libhdf5_hl.a;$HDFDIR/lib/libhdf5.a;$ZLIB_DIR/lib/libz.a" \
        -D TPL_ENABLE_Pnetcdf:STRING=ON \
        -D Pnetcdf_INCLUDE_DIRS:PATH=$NETCDF/include \
        -D Pnetcdf_LIBRARY_DIRS:PATH=$NETCDF/lib \
    \
        -D TPL_ENABLE_HDF5:STRING=ON \
        -D HDF5_INCLUDE_DIRS:PATH=$HDFDIR/include \
        -D HDF5_LIBRARY_DIRS:PATH="$HDFDIR/lib;$ZLIB_DIR/lib" \
    \
        -D TPL_ENABLE_Zlib:STRING=ON \
        -D Zlib_INCLUDE_DIRS:PATH=$ZLIB_DIR/include \
        -D Zlib_LIBRARY_DIRS:PATH=$ZLIB_DIR/lib \
    \
        -D TPL_ENABLE_BLAS:BOOL=ON \
        -D TPL_ENABLE_LAPACK:BOOL=ON \
        -D BLAS_LIBRARY_DIRS:FILEPATH=${INTEL_DIR} \
        -D TPL_BLAS_LIBRARIES:STRING="${INTEL_DIR}/libmkl_intel_lp64.a;${INTEL_DIR}/libmkl_sequential.a;${INTEL_DIR}/libmkl_core.a" \
        -D LAPACK_LIBRARY_NAMES:STRING= \
    \
        -D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \
        -D TPL_ENABLE_X11=OFF \
        -D FC_FN_CASE=LOWER \
        -D FC_FN_UNDERSCORE=UNDER  \
        -D Trilinos_ENABLE_MueLu:BOOL=OFF \
    	-D TPL_FIND_SHARED_LIBS:BOOL=OFF \
    \
    $EXTRA_ARGS \
    ${TRILINOS_HOME}
    
    exit 0
  

• Build Trilinos

    make -j 8
    make install
  

Build Albany

• Clone the repo

    cd /usr/local/src
    git clone [email protected]:SNLComputation/Albany.git
    cd Albany
  

• Configure the Albany build

    mkdir build
    cd build
    ./do-configure
  

• CMake configure script ./do-configure:

    #!/bin/sh
    
    export ALB_64BIT_INT=OFF
    export ALB_EPETRA=ON
    export ALB_SEE=OFF
    export ALB_ENABLE_SCOREC=ON
    export ALB_ENABLE_LCM=ON
    export ALB_ENABLE_LAME=ON
    export ALB_ENABLE_HYDRIDE=OFF
    export ALB_ENABLE_QCAD=OFF
    export ALB_ENABLE_MOR=OFF
    export ALB_ENABLE_ASCR=OFF
    export ALB_ENABLE_AERAS=OFF
    export ALB_ENABLE_FELIX=OFF
    export ALB_ENABLE_AlbanyCI=OFF
    export ALB_DEBUG=OFF
    export ALB_ENABLE_CHECK_FPE=OFF
    export ALB_ENABLE_SPECULATIVE=OFF
    export ALB_ENABLE_SG_MP=OFF
    export ALB_ENABLE_PERF_TESTS=ON
    export TRILINOS_INSTALL_DIR=/usr/local/host/trilinos/MPI_MIC
    export ALB_VERBOSE=OFF
    
    cmake \
          -D ALBANY_TRILINOS_DIR:FILEPATH="$TRILINOS_INSTALL_DIR" \
          -D ENABLE_LCM:BOOL=${ALB_ENABLE_LCM} \
          -D ENABLE_AERAS:BOOL=${ALB_ENABLE_AERAS} \
          -D ENABLE_QCAD:BOOL=${ALB_ENABLE_QCAD} \
          -D ENABLE_HYDRIDE:BOOL=${ALB_ENABLE_HYDRIDE} \
          -D ENABLE_LCM_SPECULATIVE:BOOL=${ALB_ENABLE_SPECULATIVE} \
          -D ENABLE_LAME:BOOL=${ALB_ENABLE_LAME} \
          -D CMAKE_VERBOSE_MAKEFILE:BOOL=${ALB_VERBOSE} \
          -D ENABLE_DEBUGGING:BOOL=${ALB_DEBUG} \
          -D ENABLE_CHECK_FPE:BOOL=${ALB_ENABLE_CHECK_FPE} \
          -D ENABLE_SCOREC:BOOL=${ALB_ENABLE_SCOREC} \
          -D ENABLE_FELIX:BOOL=${ALB_ENABLE_FELIX} \
          -D ENABLE_MOR:BOOL=${ALB_ENABLE_MOR} \
          -D ENABLE_ALBANY_CI:BOOL=${ALB_ENABLE_AlbanyCI} \
          -D ENABLE_ASCR:BOOL=${ALB_ENABLE_ASCR} \
          -D ENABLE_SG_MP:BOOL=${ALB_ENABLE_SG_MP} \
          -D ENABLE_PERFORMANCE_TESTS:BOOL=${ALB_ENABLE_PERF_TESTS} \
          -D ALBANY_CTEST_TIMEOUT:INTEGER=70 \
          -D ENABLE_64BIT_INT:BOOL=${ALB_64BIT_INT} \
          -D ENABLE_ALBANY_EPETRA_EXE:BOOL=${ALB_EPETRA} \
          -D ENABLE_SEE:BOOL=${ALB_SEE} \
          -D ALBANY_CXX_FLAGS:STRING=${ALB_FLAGS} \
    	  -D ENABLE_KOKKOS_UNDER_DEVELOPMENT:BOOL=ON \
    	  -D ENABLE_CROSS_COMPILE:BOOL=ON \
    	  -D ALBANY_MPI_OPTIONS:BOOL=ON \
    	  -D ALBANY_MPI_EXEC:STRING=${MPI_BASE_DIR}/bin/mpiexec.hydra \
    	  -D ALBANY_MPI_EXEC_NUMPROCS_FLAG:STRING=-n \
    	  -D ALBANY_MPI_EXEC_MAX_NUMPROCS:STRING=4 \
    	  -D ALBANY_MPI_TRAILING_OPTIONS:STRING='-hosts mic1 -ppn 4 -env OMP_NUM_THREADS 56 -env KMP_AFFINITY balanced -binding domain=omp -env LD_LIBRARY_PATH ${INTEL_DIR}' \
    \
    ..
  

• Build Albany

    make -j 8
    make install
  

• Test Albany

    ctest