Skip to content

System Specific Notes

Jump to bottom Edit New page
Benjamin S. Kirk edited this page May 27, 2013 · 21 revisions

Notes for Installing libMesh on specific computers

pleiades.nas.nasa.gov

Preliminaries, environment, & modules

I run bash and the important bits of my ~/.profile are

module purge
module load git/1.7.7.4
module load comp-intel/2012.0.032
module load mpi-sgi/mpt.2.06rp16
module load boost
module load svn/1.6.11

# tell MPT about our preferred compilers
export MPICC_CC=icc
export MPICXX_CXX=icpc
export MPIF90_F90=ifort

Getting PETSc

Download and build PETSc:

cd /nobackup/$USER
mkdir -p codes/petsc
cd codes/petsc
wget http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.3-p5.tar.gz
tar zxvf petsc-3.3-p5.tar.gz
mv petsc-3.3-p5 3.3-p5
cd 3.3-p5
export PETSC_DIR=`pwd`
export PETSC_ARCH=icc-12.1.4-mpt-2.06rp16
./configure --with-cxx=`which mpicxx` --with-cc=`which mpicc` --with-fc=`which mpif90` \
            --with-shared-libraries --with-debugging=0 \
            --with-blas-lapack-lib="-L/nasa/intel/Compiler/2012.0.032/mkl/lib/intel64 -Wl,-rpath,/nasa/intel/Compiler/2012.0.032/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core" \
            --download-hypre=yes --with-openmp=1
make PETSC_DIR=/nobackup/bkirk/codes/petsc/3.3-p5 PETSC_ARCH=icc-12.1.4-mpt-2.06rp16 all

Getting libMesh

Check out libMesh from GitHub:

cd /nobackup/$USER
mkdir -p codes
cd codes
git clone https://github.com/libMesh/libmesh.git
cd libmesh

configure libMesh

Configure libMesh for this platform. Make sure that your PETSC_DIR and PETSC_ARCH environment variables are set from the step above. Note that here we pass optional arguments to configure to help it find Intel's Threading Building Blocks, which are not required to build libMesh but worth having since they are available.

./configure --prefix=/nobackup/$USER/codes/install \
            --with-tbb=/nasa/intel/Compiler/2012.0.032/tbb \
            --with-tbb-lib=/nasa/intel/Compiler/2012.0.032/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21
make -j 6
make install

redwood-2.arc.nasa.gov

Preliminaries, environment, & modules

Use the intel-12.1 compiler with openmpi-1.4.4:

module load compilers/intel/intel-12
module load mpi/openmpi-1.4.4-intel64

Getting PETSc

Download and build PETSc:

wget http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.3-p5.tar.gz
tar zxvf petsc-3.3-p5.tar.gz
cd petsc-3.3-p5
export PETSC_DIR=`pwd`
export PETSC_ARCH=icc-12.1-openmpi-1.4
./configure --with-cxx=`which mpicxx` --with-cc=`which mpicc` --with-fc=`which mpif90` \
            --with-shared-libraries --with-debugging=0 \
            --with-blas-lapack-lib="-L/share/apps/Intel/composer_xe_2011_sp1.7.256/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -Wl,-rpath,/share/apps/Intel/composer_xe_2011_sp1.7.256/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core" \
            --download-hypre=yes --with-openmp=1
make PETSC_DIR=/home/gepalmer/DPLR_CHAR/PETSC/petsc-3.3-p5 PETSC_ARCH=icc-12.1-openmpi-1.4 all

configure libMesh

Configure libMesh for this platform. Make sure that your PETSC_DIR and PETSC_ARCH environment variables are set from the step above. Note that here we pass optional arguments to configure to help it find Intel's Threading Building Blocks, which are not required to build libMesh but worth having since they are available.

./configure --prefix=`pwd`/install \
            --with-tbb=/share/apps/Intel/composer_xe_2011_sp1.7.256/composer_xe_2011_sp1.7.256/tbb \
            --with-tbb-lib=/share/apps/Intel/composer_xe_2011_sp1.7.256/composer_xe_2011_sp1.7.256/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21
make -j 8
make install
# this next line should probably be in the intel compiler module file!!
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/share/apps/Intel/composer_xe_2011_sp1.7.256/composer_xe_2011_sp1.7.256/tbb/lib/intel64/cc4.1.0_libc2.4_kernel2.6.16.21 
make -j 8 check LIBMESH_RUN="mpirun -np 2" METHODS="opt"

stampede.tacc.utexas.edu

Preliminaries, environment, & modules

My modules:

login4$ module list
Currently Loaded Modules:
  1) TACC          4) cluster             7) mvapich2/1.9a2      10) git/1.8.1.1      13) hdf5/1.8.9
  2) TACC-paths    5) cluster-paths       8) trilinos/10.12.2    11) slepc/3.3-cxx
  3) Linux         6) intel/13.0.2.146    9) boost/1.51.0        12) petsc/3.3-cxx

Getting libMesh

Check out libMesh from GitHub:

mkdir -p $WORK/codes
cd $WORK/codes
git clone https://github.com/libMesh/libmesh.git
cd libmesh
mkdir -p build && cd build
../configure \
  --prefix=$WORK/codes/install \
  --with-boost=$TACC_BOOST_DIR \
  --with-trilinos=$TACC_TRILINOS_DIR \
  --with-tbb=/opt/apps/intel/13/composer_xe_2013.2.146/tbb \
  --with-tbb-lib=/opt/apps/intel/13/composer_xe_2013.2.146/tbb/lib/intel64 \
  --with-hdf5=$TACC_HDF5_DIR \
  METHODS="devel opt"
make -j 6 && rm -rf $WORK/codes/install && make install
cd $WORK/codes/install/examples/introduction/ex4
make

Now, test the code:

srun -A nasa-cap -p development -t 01:00:00 -n16 --pty /bin/bash -l
export LD_LIBRARY_PATH=$TACC_TRILINOS_LIB:$LD_LIBRARY_PATH 
ibrun example-opt -d 3 -n 50
ibrun example-opt -d 3 -n 50 --use-trilinos

plx.cineca.it

Preliminaries, environment, & modules

I run bash and the important bits of my ~/.bash_profile are

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

# load preferred modules
module load intel/11.1--binary
module load openmpi/1.4.2--intel--11.1--binary
module load cmake/2.8.8 valgrind

unset CXX CC FC F77 F90

export ARCH_ID_STRING=intel-11.1-openmpi-1.4

export CODES=$HOME/codes
export PETSC_DIR=$CODES/petsc/3.4.0
export PETSC_ARCH=$ARCH_ID_STRING
export TRILINOS_DIR=$CODES/trilinos/11.2.2/$ARCH_ID_STRING
export TBB_DIR=$INTEL_HOME/tbb

PATH=$CODES/libmesh/contrib/autotools/bin:$PATH

export PATH

# aliases
alias make='make -s --no-print-directory'
alias ls='ls --color'

# Matlab-style history
bind '"\e[A"':history-search-backward
bind '"\e[B"':history-search-forward

Getting PETSc

Download and build PETSc:

cd /nobackup/$USER
mkdir -p codes/petsc
cd codes/petsc

wget http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-lite-3.3-p7.tar.gz
tar zxf petsc-lite-3.3-p7.tar.gz 
mv petsc-3.3-p7 3.3-p7
cd 3.3-p7
export PETSC_DIR=`pwd`
export PETSC_ARCH=intel-11.1-openmpi-1.4
./configure --with-cxx="`which mpicxx` -openmp" --with-cc="`which mpicc` -openmp" --with-fc="`which mpif90` -openmp" \
            --with-shared-libraries --with-debugging=0 \
            --with-blas-lapack-lib="-L$INTEL_HOME/mkl/lib/em64t -Wl,-rpath,$INTEL_HOME/mkl/lib/em64t -lmkl_intel_lp64 -lmkl_sequential -lmkl_core" \
            --download-hypre=yes --with-openmp=1
make PETSC_DIR=/plx/usera06trn/a06trn26/codes/petsc/3.3-p7 PETSC_ARCH=intel-11.1-openmpi-1.4 all

Getting Trilinos

cmake -DCMAKE_BUILD_TYPE:STRING=RELEASE \
      -DTPL_ENABLE_MKL:BOOL=ON -DMKL_INCLUDE_DIRS:PATH="$INTEL_HOME/mkl/include" \
      -DMKL_LIBRARY_DIRS:PATH="$INTEL_HOME/mkl/lib/em64t" -DBLAS_LIBRARY_DIRS:PATH="$INTEL_HOME/mkl/lib/em64t" \
      -DBLAS_LIBRARY_NAMES:STRING="mkl_intel_lp64;mkl_sequential;mkl_core" \
      -DLAPACK_LIBRARY_DIRS:PATH="$INTEL_HOME/mkl/lib/em64t" \
      -DLAPACK_LIBRARY_NAMES:STRING="mkl_intel_lp64;mkl_sequential;mkl_core" \
      -DTrilinos_ENABLE_OpenMP:BOOL=ON -DTrilinos_ENABLE_TESTS:BOOL=ON -DTrilinos_ENABLE_Tpetra=ON \
      -DTrilinos_ENABLE_Epetra=ON -DTrilinos_ENABLE_NOX=ON -DTrilinos_ENABLE_AztecOO=ON \
      -DTrilinos_ENABLE_ML=ON -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON \
      -DTrilinos_ENABLE_TESTS:BOOL=ON -DTPL_ENABLE_MPI:BOOL=ON -DMPI_BASE_DIR:PATH=$OMPI_HOME \
      -DMPI_BIN_DIR:PATH=$OMPI_HOME/bin -DBUILD_SHARED_LIBS:BOOL=ON \
      -DMPI_EXEC:FILEPATH="$OMPI_HOME/bin/mpiexec"  -DCMAKE_C_COMPILER:FILEPATH="$OMPI_HOME/bin/mpicc" \
      -DCMAKE_CXX_COMPILER:FILEPATH="$OMPI_HOME/bin/mpicxx" \
      -DCMAKE_Fortran_COMPILER:FILEPATH="$OMPI_HOME/bin/mpif90" \
      -DCMAKE_INSTALL_PREFIX:PATH=$TRILINOS_DIR ${TRILINOS_HOME}

libMesh

./configure --prefix=$CODES/install \
            --with-tbb=$TBB_DIR \
            --with-tbb-lib="$TBB_DIR/lib/intel64 -Wl,-rpath,$TBB_DIR/lib/intel64"
Add a custom footer

Navigation

Clone this wiki locally