update setonix configuration #362

BenWibking · 2023-09-06T16:38:21Z

Update the Setonix build and runtime configuration following the September 2023 upgrade.

BenWibking · 2023-09-06T16:41:51Z

None of the modified files are used by the CI, so it should not be necessary to run the GPU tests for this PR.

BenWibking · 2023-09-06T16:46:14Z

(Somehow I created something unmergable on the old branch, so this is a new PR.)

BenWibking · 2023-09-07T15:21:41Z

@psharda Can you see if this works for you?

BenWibking · 2023-09-07T23:39:44Z

ping @psharda

psharda · 2023-09-08T00:59:24Z

ah sorry , just saw this now

psharda · 2023-09-08T01:06:26Z

I merged this branch in my popiii branch. I get some warnings that I didnt get before on Setonix:

psharda@setonix-01:/software/projects/pawsey0807/psharda/quokka/build> cmake -C ../cmake/setonix.cmake  ..
loading initial cache file ../cmake/setonix.cmake
-- The CXX compiler identification is Clang 15.0.6
-- The C compiler identification is Clang 15.0.6
-- Cray Programming Environment 2.7.20 CXX
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/cray/pe/craype/2.7.20/bin/CC - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Cray Programming Environment 2.7.20 C
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/cray/pe/craype/2.7.20/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- CMake version: 3.24.3
-- AMReX installation directory: /usr/local
-- Building AMReX with AMReX_SPACEDIM = 3
-- Configuring AMReX with the following options enabled:
--    AMReX_PRECISION = DOUBLE
--    AMReX_GPU_BACKEND = HIP
--    AMReX_GPU_MAX_THREADS = 256
--    AMReX_GPU_RDC
--    AMReX_MPI
--    AMReX_LINEAR_SOLVERS
--    AMReX_PARTICLES
--    AMReX_PARTICLES_PRECISION = DOUBLE
--    AMReX_CONDUIT
--    AMReX_ASCENT
--    AMReX_TINY_PROFILE
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found MPI_C: /opt/cray/pe/craype/2.7.20/bin/cc (found version "3.1")
-- Found MPI_CXX: /opt/cray/pe/craype/2.7.20/bin/CC (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: C CXX
-- hip::amdhip64 is SHARED_LIBRARY
-- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS
-- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS - Failed
-- Found HIP:
-- HIP: Runtime=rocclr Compiler=clang Path=/opt/rocm/hip
-- hip::amdhip64 is SHARED_LIBRARY
-- hip::amdhip64 is SHARED_LIBRARY
-- hip::amdhip64 is SHARED_LIBRARY
-- Performing Test LINKER_HAS_WHOLE_ARCHIVE_OFFLOAD
-- Performing Test LINKER_HAS_WHOLE_ARCHIVE_OFFLOAD - Success
-- hip::amdhip64 is SHARED_LIBRARY
-- Conduit was built with Zlib Support
-- Looking for Zlib at: /software/projects/pawsey0807/bwibking/ascent_06082023/install/zlib-1.2.13/
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11")
-- Conduit was built with HDF5 Support
-- Looking for HDF5 at: /opt/cray/pe/hdf5/1.12.2.3/crayclang/14.0
-- Found HDF5: Included by compiler wrappers (found version "1.12.2")
-- HDF5_DIR_REAL=/opt/cray/pe/hdf5/1.12.2.3/crayclang/14.0
-- Checking that found HDF5_INCLUDE_DIRS are in HDF5_DIR
CMake Warning at /software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/lib/cmake/conduit/conduit_setup_deps.cmake:181 (message):
  FindHDF5 did not provide HDF5_INCLUDE_DIRS or HDF5_INCLUDE_DIR.
Call Stack (most recent call first):
  /software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/lib/cmake/conduit/ConduitConfig.cmake:76 (include)
  /software/setonix/2023.08/software/linux-sles15-zen3/cce-15.0.1/cmake-3.24.3-p6jl2y6yn7ho4t5v74ejibshhscmehtz/share/cmake-3.24/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /software/projects/pawsey0807/bwibking/ascent_06082023/install/ascent-develop/lib/cmake/ascent/ascent_setup_deps.cmake:61 (find_dependency)
  /software/projects/pawsey0807/bwibking/ascent_06082023/install/ascent-develop/lib/cmake/ascent/AscentConfig.cmake:150 (include)
  extern/amrex/Tools/CMake/AMReXThirdPartyLibraries.cmake:83 (find_package)
  extern/amrex/Src/CMakeLists.txt:50 (include)


-- HDF5_INCLUDE_DIRS=
CMake Warning at /software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/lib/cmake/conduit/conduit_setup_deps.cmake:216 (message):
  HDF5_INCLUDE_DIRS () does not include HDF5_DIR
Call Stack (most recent call first):
  /software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/lib/cmake/conduit/ConduitConfig.cmake:76 (include)
  /software/setonix/2023.08/software/linux-sles15-zen3/cce-15.0.1/cmake-3.24.3-p6jl2y6yn7ho4t5v74ejibshhscmehtz/share/cmake-3.24/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /software/projects/pawsey0807/bwibking/ascent_06082023/install/ascent-develop/lib/cmake/ascent/ascent_setup_deps.cmake:61 (find_dependency)
  /software/projects/pawsey0807/bwibking/ascent_06082023/install/ascent-develop/lib/cmake/ascent/AscentConfig.cmake:150 (include)
  extern/amrex/Tools/CMake/AMReXThirdPartyLibraries.cmake:83 (find_package)
  extern/amrex/Src/CMakeLists.txt:50 (include)


-- HDF5 is parallel:  FALSE
-- Found Conduit: /software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8 (found version 0.8.8)
-- CONDUIT_VERSION             = 0.8.8
-- CONDUIT_INSTALL_PREFIX      = /software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8
-- CONDUIT_IMPORT_ROOT         = /software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8
-- CONDUIT_USE_CXX11           = TRUE
-- CONDUIT_USE_FMT             = TRUE
-- CONDUIT_INCLUDE_DIRS        = /software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/include/conduit
-- CONDUIT_FORTRAN_ENABLED     = FALSE
-- CONDUIT_PYTHON_ENABLED      =
-- CONDUIT_PYTHON_EXECUTABLE   =
-- CONDUIT_PYTHON_MODULE_DIR   = /software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/python-modules/
-- Conduit Relay features:
--  CONDUIT_RELAY_WEBSERVER_ENABLED = TRUE
--  CONDUIT_RELAY_HDF5_ENABLED      = TRUE
--  CONDUIT_HDF5_DIR                = /opt/cray/pe/hdf5/1.12.2.3/crayclang/14.0
--  CONDUIT_RELAY_ADIOS_ENABLED     = FALSE
--  CONDUIT_ADIOS_DIR               =
--  CONDUIT_RELAY_SILO_ENABLED      = FALSE
--  CONDUIT_SILO_DIR                =
--  CONDUIT_RELAY_MPI_ENABLED       = TRUE
-- Conduit imported targets: conduit::conduit conduit::conduit_mpi
-- Enabled Kokkos devices: SERIAL;HIP
-- Enabled Kokkos devices: SERIAL;HIP
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = "",
	LC_CTYPE = "UTF-8",
	LC_MESSAGES = "",
	LANG = ""
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = "",
	LC_CTYPE = "UTF-8",
	LC_MESSAGES = "",
	LANG = ""
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
-- The HIP compiler identification is Clang 14.0.0
-- Detecting HIP compiler ABI info
-- Detecting HIP compiler ABI info - done
-- Check for working HIP compiler: /opt/rocm/llvm/bin/clang++ - skipped
-- Detecting HIP compile features
-- Detecting HIP compile features - done
-- ASCENT_VERSION             = 0.9.2
-- ASCENT_INSTALL_PREFIX      = /software/projects/pawsey0807/bwibking/ascent_06082023/install/ascent-develop
-- ASCENT_INCLUDE_DIRS        = /software/projects/pawsey0807/bwibking/ascent_06082023/install/ascent-develop/include/ascent
-- ASCENT_FORTRAN_ENABLED     = OFF
-- ASCENT_PYTHON_ENABLED      =
-- ASCENT_PYTHON_EXECUTABLE   =
-- ASCENT_SERIAL_ENABLED      = ON
-- ASCENT_MPI_ENABLED         = ON
-- ASCENT_CUDA_ENABLED        = OFF
-- ASCENT_HIP_ENABLED         = ON
-- ASCENT_OPENMP_ENABLED      = OFF
-- ASCENT_VTKH_ENABLED        = ON
-- ASCENT_CAMP_ENABLED        = TRUE
-- ASCENT_UMPIRE_ENABLED      = TRUE
-- ASCENT_RAJA_ENABLED        = 1
-- ASCENT_DRAY_ENABLED        = ON
-- ASCENT_APCOMP_ENABLED      = ON
-- ASCENT_OCCA_ENABLED        =
-- ASCENT_BABELFLOW_ENABLED   =
-- ASCENT_FIDES_ENABLED       =
-- ASCENT_MFEM_ENABLED        = TRUE
-- ASCENT_MFEM_MPI_ENABLED    = FALSE
-- Ascent imported targets: ascent::ascent ascent::ascent_mpi
-- AMReX configuration summary:
--    Build type               = Release
--    Install directory        = /usr/local
--    C++ compiler             = /opt/cray/pe/craype/2.7.20/bin/CC
--    C++ defines              = -DAMREX_SPACEDIM=3 -D__HIP_PLATFORM_HCC__=1 -D__HIP_PLATFORM_AMD__=1 -DCAMP_HAVE_HIP -DVTKMDIY_NO_THREADS -DVTKMDIY_MPI_AS_LIB
--    C++ flags                = -O3 -DNDEBUG -I/opt/rocm/include -mllvm -amdgpu-early-inline-all=true -amdgpu-function-calls=false -m64 -munsafe-fp-atomics -std=c++17 -fgpu-rdc -x hip --offload-arch=gfx90a -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/include/conduit -DCONDUIT_RELAY_IO_MPI_ENABLED -DCONDUIT_BLUEPRINT_MPI_ENABLED --rocm-path=/opt/rocm -ffunction-sections -DROVER_PARALLEL
--    C++ include paths        = -I/software/projects/pawsey0807/psharda/quokka/extern/amrex/Src/Base -I/software/projects/pawsey0807/psharda/quokka/extern/amrex/Src/Base/Parser -I/software/projects/pawsey0807/psharda/quokka/extern/amrex/Src/Boundary -I/software/projects/pawsey0807/psharda/quokka/extern/amrex/Src/AmrCore -I/software/projects/pawsey0807/psharda/quokka/extern/amrex/Src/LinearSolvers/MLMG -I/software/projects/pawsey0807/psharda/quokka/extern/amrex/Src/LinearSolvers/OpenBC -I/software/projects/pawsey0807/psharda/quokka/extern/amrex/Src/Particle -I/software/projects/pawsey0807/psharda/quokka/extern/amrex/Src/Extern/Conduit -I/opt/rocm-5.2.3/include -I/opt/rocm/llvm/lib/clang/14.0.0/include/.. -I/opt/rocm-5.2.3/include/hiprand -I/opt/rocm-5.2.3/include/rocrand -I/opt/rocm-5.2.3/include/ -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/ascent-develop/include/ -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/ascent-develop/include/ascent/ -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/include/ -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/include/conduit/ -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/mfem-4.5.2/include -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/include/conduit -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/raja-v2022.10.4/include -I/opt/rocm/include -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/camp-2022.10.1/include -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/umpire-2022.10.0/include -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/vtk-m-v2.1.0/include/vtkm-2.1/vtkm/thirdparty/optionparser -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/vtk-m-v2.1.0/include/vtkm-2.1/vtkm/thirdparty/diy -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/vtk-m-v2.1.0/include/vtkm-2.1/vtkm/thirdparty/diy/vtkmdiy/include -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/vtk-m-v2.1.0/include/vtkm-2.1/vtkm/thirdparty/diy/vtkmdiy/include/vtkmdiy/mpi -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/vtk-m-v2.1.0/include/vtkm-2.1/vtkm/thirdparty/lcl/vtkmlcl -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/vtk-m-v2.1.0/include/vtkm-2.1 -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/kokkos-3.7.02/include -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/ascent-develop/include/ascent/vtkh -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/ascent-develop/include/ascent/dray -I/software/projects/pawsey0807/bwibking/ascent_06082023/install/ascent-develop/include/ascent/apcomp
--    Link line                = /opt/rocm/llvm/lib/clang/14.0.0/lib/linux/libclang_rt.builtins-x86_64.a dl rt -lz -ldl -lm /software/projects/pawsey0807/bwibking/spack/opt/spack/linux-sles15-zen/gcc-7.5.0/zlib-ng-2.1.3-b3awwnssxz32ys4ar42yuifhlab6bb4p/lib/libz.so /software/projects/pawsey0807/bwibking/ascent_06082023/install/mfem-4.5.2/lib/libmfem.so -Wl,-rpath,/software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/lib -L/software/projects/pawsey0807/bwibking/ascent_06082023/install/conduit-v0.8.8/lib -lconduit -lconduit_relay -lconduit_blueprint /usr/lib64/libz.so
-- Version: 10.1.1
-- Build type: Release
-- Performing Test HAS_NULLPTR_WARNING
-- Performing Test HAS_NULLPTR_WARNING - Success
-- Found Python: /opt/cray/pe/python/3.9.13.1/bin/python3.9 (found version "3.9.13") found components: Interpreter Development Development.Module Development.Embed
Building and linking amrex -- false
Not building and fetching amrex in Microphysics
Building unit test -- false
Not building primordial chem unit test
-- Building Quokka *without* Python support
-- Fused multiply-add (FMAD) is disabled for device code.

write_probin.py: creating extern parameter files
write_probin.py: working on parameter file /software/projects/pawsey0807/psharda/quokka/extern/Microphysics/EOS/gamma_law/_parameters...
write_probin.py: working on parameter file /software/projects/pawsey0807/psharda/quokka/extern/Microphysics/networks/_parameters...
write_probin.py: working on parameter file /software/projects/pawsey0807/psharda/quokka/extern/Microphysics/integration/VODE/_parameters...
write_probin.py: working on parameter file /software/projects/pawsey0807/psharda/quokka/extern/Microphysics/integration/_parameters...
write_network.py: working on network file /software/projects/pawsey0807/psharda/quokka/extern/Microphysics/networks/general_null/gammalaw.net ...
writing /software/projects/pawsey0807/psharda/quokka/build/src//network_properties.H


write_probin.py: creating extern parameter files
write_probin.py: working on parameter file /software/projects/pawsey0807/psharda/quokka/extern/Microphysics/EOS/primordial_chem/_parameters...
write_probin.py: working on parameter file /software/projects/pawsey0807/psharda/quokka/extern/Microphysics/networks/primordial_chem/_parameters...
write_probin.py: working on parameter file /software/projects/pawsey0807/psharda/quokka/extern/Microphysics/networks/_parameters...
write_probin.py: working on parameter file /software/projects/pawsey0807/psharda/quokka/extern/Microphysics/integration/VODE/_parameters...
write_probin.py: working on parameter file /software/projects/pawsey0807/psharda/quokka/extern/Microphysics/integration/_parameters...
0
write_network.py: working on network file /software/projects/pawsey0807/psharda/quokka/extern/Microphysics/networks/primordial_chem/pynucastro.net ...
writing /software/projects/pawsey0807/psharda/quokka/build/src/PopIII//network_properties.H

-- Configuring done
CMake Warning at src/PopIII/CMakeLists.txt:12 (add_executable):
  Cannot generate a safe runtime search path for target popiii because files
  in some directories may conflict with libraries in implicit directories:

    runtime library [libz.so.1] in /usr/lib64 may be hidden by files in:
      /software/projects/pawsey0807/bwibking/spack/opt/spack/linux-sles15-zen/gcc-7.5.0/zlib-ng-2.1.3-b3awwnssxz32ys4ar42yuifhlab6bb4p/lib

  Some of these libraries may not be found correctly.


-- Generating done
-- Build files have been written to: /software/projects/pawsey0807/psharda/quokka/build

BenWibking · 2023-09-08T01:07:22Z

All of the warnings are coming from the Ascent build. If you turn off Ascent, they should go away.

psharda · 2023-09-08T01:08:57Z

So then should ASCENT be OFF by default in setonix.cmake?

BenWibking · 2023-09-08T01:15:08Z

I think it should be on by default, unless you don't think you'll want to run with it on?

BenWibking · 2023-09-08T01:36:50Z

So then should ASCENT be OFF by default in setonix.cmake?

Since Ascent is OFF by default, I've commented out that line.

Users can un-comment or set -DAMReX_ASCENT=ON -DAMReX_CONDUIT=ON on the command line in order to turn it on.

psharda · 2023-09-08T01:54:05Z

/azp run

scripts/setonix.profile

psharda · 2023-09-08T17:51:22Z

/azp run

psharda · 2023-09-08T18:57:55Z

@BenWibking seems that we need > 60 minutes on avatargpus now?

BenWibking · 2023-09-08T19:00:28Z

@BenWibking seems that we need > 60 minutes on avatargpus now?

hmm it does look like it. can you increase the timeout to, say, 90 minutes?

psharda · 2023-09-08T19:02:24Z

oh but it's your branch

BenWibking · 2023-09-08T19:03:36Z

oh but it's your branch

ah lol, oops. too many PRs, sorry.

psharda · 2023-09-09T04:45:08Z

/azp run

psharda · 2023-09-09T14:11:53Z

/azp run

BenWibking · 2023-09-09T19:59:17Z

@psharda can you approve?

psharda · 2023-09-09T21:42:18Z

Approved from 37000 ft above sea level

BenWibking added 2 commits September 5, 2023 21:57

update setonix configuration

2a119df

update config

62dc1a3

BenWibking requested a review from psharda September 6, 2023 16:38

Merge branch 'development' into update-setonix-config-new

1c48d1a

BenWibking enabled auto-merge September 6, 2023 16:39

use un-hidden cmake

c160498

BenWibking changed the title ~~Update setonix config~~ update setonix configuration Sep 7, 2023

Merge branch 'development' into update-setonix-config-new

d70b66d

Update setonix.cmake

85557c9

Merge branch 'development' into update-setonix-config-new

6ed72bb

psharda reviewed Sep 8, 2023

View reviewed changes

scripts/setonix.profile Show resolved Hide resolved

Merge branch 'development' into update-setonix-config-new

b4acebb

Merge branch 'development' into update-setonix-config-new

f52c66c

Merge branch 'development' into update-setonix-config-new

37aa2d5

psharda approved these changes Sep 9, 2023

View reviewed changes

BenWibking added this pull request to the merge queue Sep 9, 2023

Merged via the queue into development with commit d17b5e4 Sep 9, 2023
8 checks passed

BenWibking deleted the update-setonix-config-new branch September 10, 2023 00:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update setonix configuration #362

update setonix configuration #362

BenWibking commented Sep 6, 2023

BenWibking commented Sep 6, 2023

BenWibking commented Sep 6, 2023

BenWibking commented Sep 7, 2023

BenWibking commented Sep 7, 2023

psharda commented Sep 8, 2023

psharda commented Sep 8, 2023

BenWibking commented Sep 8, 2023

psharda commented Sep 8, 2023

BenWibking commented Sep 8, 2023

BenWibking commented Sep 8, 2023 •

edited

Loading

psharda commented Sep 8, 2023

psharda commented Sep 8, 2023

psharda commented Sep 8, 2023

BenWibking commented Sep 8, 2023

psharda commented Sep 8, 2023

BenWibking commented Sep 8, 2023

psharda commented Sep 9, 2023

psharda commented Sep 9, 2023

BenWibking commented Sep 9, 2023

psharda commented Sep 9, 2023

update setonix configuration #362

update setonix configuration #362

Conversation

BenWibking commented Sep 6, 2023

BenWibking commented Sep 6, 2023

BenWibking commented Sep 6, 2023

BenWibking commented Sep 7, 2023

BenWibking commented Sep 7, 2023

psharda commented Sep 8, 2023

psharda commented Sep 8, 2023

BenWibking commented Sep 8, 2023

psharda commented Sep 8, 2023

BenWibking commented Sep 8, 2023

BenWibking commented Sep 8, 2023 • edited Loading

psharda commented Sep 8, 2023

psharda commented Sep 8, 2023

psharda commented Sep 8, 2023

BenWibking commented Sep 8, 2023

psharda commented Sep 8, 2023

BenWibking commented Sep 8, 2023

psharda commented Sep 9, 2023

psharda commented Sep 9, 2023

BenWibking commented Sep 9, 2023

psharda commented Sep 9, 2023

BenWibking commented Sep 8, 2023 •

edited

Loading