Skip to content

Commit

Permalink
Move installation instructions to docs (#607)
Browse files Browse the repository at this point in the history 
### Description

Moved installation instructions from README to docs and added a link to
the docs in README.md

### Related issues
#605 

### Checklist
_Before this pull request can be reviewed, all of these tasks should be
completed. Denote completed tasks with an `x` inside the square brackets
`[ ]` in the Markdown source below:_
- [x] I have added a description (see above).
- [x] I have added a link to any related issues see (see above).
- [x] I have read the [Contributing
Guide](https://github.com/quokka-astro/quokka/blob/development/CONTRIBUTING.md).
- [ ] I have added tests for any new physics that this PR adds to the
code.
- [ ] I have tested this PR on my local computer and all tests pass.
- [ ] I have manually triggered the GPU tests with the magic comment
`/azp run`.
- [x] I have requested a reviewer for this PR.
  • Loading branch information
@chongchonghe
chongchonghe authored Apr 14, 2024
1 parent a49bfe1 commit f4784d2
Show file tree
Hide file tree
Showing 2 changed files with 139 additions and 120 deletions.
104 changes: 1 addition & 103 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

**The Quokka methods paper is now available: https://arxiv.org/abs/2110.01792**

**Please see the sections "Quickstart" and "Running on GPUs" below. You can start a [Discussion](https://github.com/BenWibking/quokka/discussions) for technical support, or open an [Issue](https://github.com/BenWibking/quokka/issues) for any bug reports.**
**For detailed instructions on installing the code, please refer to the [Quokka Documentation](https://quokka-astro.github.io/quokka/index.html). You can start a [Discussion](https://github.com/BenWibking/quokka/discussions) for technical support, or open an [Issue](https://github.com/BenWibking/quokka/issues) for any bug reports.**

Quokka is a two-moment radiation hydrodynamics code that uses the piecewise-parabolic method, with AMR and subcycling in time. Runs on CPUs (MPI+vectorized) or NVIDIA GPUs (MPI+CUDA) with a single-source codebase. Written in C++17. (100% Fortran-free.)

Expand All @@ -36,107 +36,5 @@ Quokka also features advanced Adaptive Quokka Refinement:tm: technology:
* Python 3.7+ (optional)
* ADIOS2 2.9+ with GPU-aware support (optional, for writing terabyte-sized or larger outputs)

## Quickstart

To run Quokka, download this repository to your local machine:
```
git clone --recursive https://github.com/BenWibking/quokka.git
```
Quokka uses CMake (and optionally, Ninja) as its build system. If you don't have CMake and Ninja installed, the easiest way to install them is to run:
```
python3 -m pip install cmake ninja --user
```
Now that CMake is installed, create a `build/` subdirectory and compile Quokka, as shown below.
```
cd quokka
mkdir build; cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -G Ninja
ninja -j6
```
Congratuations! You have now built all of the 1D test problems on CPU. You can run the automated test suite:
```
ninja test
```
You should see output that indicates all tests have passed, like this:
```
100% tests passed, 0 tests failed out of 20
Total Test time (real) = 111.74 sec
```
To run in 2D or 3D, build with the `-DAMReX_SPACEDIM` CMake option, for example:
```
cmake .. -DCMAKE_BUILD_TYPE=Release -DAMReX_SPACEDIM=3 -G Ninja
ninja -j6
```
to compile Quokka for 3D problems.

**By default, Quokka compiles itself *only* for CPUs. If you want to run Quokka on GPUs, see the section "Running on GPUs" below.**

Have fun!

## Building with CMake + `make`
If you are unable to install Ninja, you can instead use CMake with the Makefile generator, which should produce identical results but is slower:
```
cmake .. -DCMAKE_BUILD_TYPE=Release -G "Unix Makefiles"
make -j6
make test
```

## Could NOT find Python error
If CMake prints an error saying that Python could not be found, e.g.:
```
-- Could NOT find Python (missing: Python_EXECUTABLE Python_INCLUDE_DIRS Python_LIBRARIES Python_NumPy_INCLUDE_DIRS Interpreter Development NumPy Development.Module Development.Embed)
```
you should be able to fix this by installing NumPy (and matplotlib) by running
```
python3 -m pip install numpy matplotlib --user
```
This should enable CMake to find the NumPy header files that are needed to successfully compile.

Alternatively, you can work around this problem by disabling Python support. Python and NumPy are only used to plot the results of some test problems, so this does not otherwise affect Quokka's functionality. Add the option
```
-DQUOKKA_PYTHON=OFF
```
to the CMake command-line options (or change the `QUOKKA_PYTHON` option to `OFF` in CMakeLists.txt).

## Running on GPUs
By default, Quokka compiles itself to run only on CPUs. Quokka can run on either NVIDIA, AMD, or Intel GPUs. Consult the sub-sections below for the build instructions for a given GPU vendor.

### NVIDIA GPUs
If you want to run on NVIDIA GPUs, re-build Quokka as shown below. (*CUDA >= 11.7 is required. Quokka is only supported on Volta V100 GPUs or newer models. Your MPI library **must** support CUDA-aware MPI.*)
```
cmake .. -DCMAKE_BUILD_TYPE=Release -DAMReX_GPU_BACKEND=CUDA -DAMReX_SPACEDIM=3 -G Ninja
ninja -j6
```
**All GPUs on a node must be visible from each MPI rank on the node for efficient GPU-aware MPI communication to take place via CUDA IPC.** When using the SLURM job scheduler, this means that `--gpu-bind` should be set to `none`.

The compiled test problems are in the test problem subdirectories in `build/src/`. Example scripts for running Quokka on compute clusters are in the `scripts/` subdirectory.

Note that 1D problems can run very slowly on GPUs due to a lack of sufficient parallelism. To run the test suite in a reasonable amount of time, you may wish to exclude the matter-energy exchange tests, e.g.:
```
ctest -E "MatterEnergyExchange*"
```
which should end with output similar to the following:
```
100% tests passed, 0 tests failed out of 18
Total Test time (real) = 353.77 sec
```
### AMD GPUs *(experimental, use at your own risk)*
Compile with `-DAMReX_GPU_BACKEND=HIP`. Requires ROCm 5.2.0 or newer. Your MPI library **must** support GPU-aware MPI for AMD GPUs. Quokka has been tested on MI100 and MI250X GPUs, but there are known compiler issues that affect the correctness of simulation results (see https://github.com/quokka-astro/quokka/issues/394 and https://github.com/quokka-astro/quokka/issues/447).

### Intel GPUs *(experimental, use at your own risk)*
Not tested. You can attempt this by compiling with `-DAMReX_GPU_BACKEND=SYCL`. Please start a Discussion if you encounter issues on Intel GPUs. Your MPI library **must** support GPU-aware MPI for Intel GPUs.

## Building a specific test problem
By default, all available test problems will be compiled. If you only want to build a specific problem, you can list all of the available CMake targets:
```
cmake --build . --target help
```
and then build the problem of interest:
```
ninja -j6 test_hydro3d_blast
```

## Problems?
If you run into problems, please start a [Discussion](https://github.com/BenWibking/quokka/discussions) for technical support. If you discover a bug, please let us know by opening an [Issue](https://github.com/BenWibking/quokka/issues).
155 changes: 138 additions & 17 deletions docs/installation.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,47 +3,168 @@
Installation
============

To run Quokka, download the Git repository to your local machine::
To run Quokka, download this repository and its submodules to your local machine::

git clone git@github.com:BenWibking/quokka.git
git clone --recursive https://github.com/quokka-astro/quokka.git

Then download all submodules (this downloads `AMReX` and the string-formatting library `fmt`)::
Quokka uses CMake (and optionally, Ninja) as its build system. If you don't have CMake and Ninja installed, the easiest way to install them is to run::

cd quokka
git submodule update --init
python3 -m pip install cmake ninja --user

Create a build/ subdirectory and compile Quokka::
Now that CMake is installed, create a `build/` subdirectory and compile Quokka, as shown below.

mkdir build; cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j6
::

cd quokka
mkdir build; cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -G Ninja
ninja -j6

Congratuations! You have now built all of the 1D test problems on CPU. You can run the automated test suite::

make test -j6
ninja test

You should see output that indicates all tests have passed, like this::

100% tests passed, 0 tests failed out of 20

Total Test time (real) = 111.74 sec

To run in 2D or 3D, edit the `AMReX_SPACEDIM` option in the `CMakeLists.txt` file, for example::
To run in 2D or 3D, build with the `-DAMReX_SPACEDIM` CMake option, for example:

::

set(AMReX_SPACEDIM 3 CACHE STRING "" FORCE)
cmake .. -DCMAKE_BUILD_TYPE=Release -DAMReX_SPACEDIM=3 -G Ninja
ninja -j6

to compile Quokka for 3D problems.

**By default, Quokka compiles itself only for CPUs. If you want to run
Quokka on GPUs, see the section “Running on GPUs” below.**

Have fun!

Building with CMake + ``make``
------------------------------

If you are unable to install Ninja, you can instead use CMake with the
Makefile generator, which should produce identical results but is
slower:

::

cmake .. -DCMAKE_BUILD_TYPE=Release -G "Unix Makefiles"
make -j6
make test

Could NOT find Python error
---------------------------

If CMake prints an error saying that Python could not be found, e.g.:

::

-- Could NOT find Python (missing: Python_EXECUTABLE Python_INCLUDE_DIRS Python_LIBRARIES Python_NumPy_INCLUDE_DIRS Interpreter Development NumPy Development.Module Development.Embed)

you should be able to fix this by installing NumPy (and matplotlib) by
running

::

python3 -m pip install numpy matplotlib --user

This should enable CMake to find the NumPy header files that are needed
to successfully compile.

Alternatively, you can work around this problem by disabling Python
support. Python and NumPy are only used to plot the results of some test
problems, so this does not otherwise affect Quokka’s functionality. Add
the option

::

-DQUOKKA_PYTHON=OFF

to the CMake command-line options (or change the ``QUOKKA_PYTHON``
option to ``OFF`` in CMakeLists.txt).


Running on GPUs
---------------

By default, Quokka compiles itself to run only on CPUs. If you want to run on NVIDIA GPUs, re-build Quokka with the following options::
By default, Quokka compiles itself to run only on CPUs. Quokka can run
on either NVIDIA, AMD, or Intel GPUs. Consult the sub-sections below for
the build instructions for a given GPU vendor.

NVIDIA GPUs
~~~~~~~~~~~

If you want to run on NVIDIA GPUs, re-build Quokka as shown below.
(*CUDA >= 11.7 is required. Quokka is only supported on Volta V100 GPUs
or newer models. Your MPI library* **must** *support CUDA-aware MPI.*)

::

cmake .. -DCMAKE_BUILD_TYPE=Release -DAMReX_GPU_BACKEND=CUDA -DAMReX_SPACEDIM=3 -G Ninja
ninja -j6

**All GPUs on a node must be visible from each MPI rank on the node for
efficient GPU-aware MPI communication to take place via CUDA IPC.** When
using the SLURM job scheduler, this means that ``--gpu-bind`` should be
set to ``none``.

The compiled test problems are in the test problem subdirectories in
``build/src/``. Example scripts for running Quokka on compute clusters
are in the ``scripts/`` subdirectory.

Note that 1D problems can run very slowly on GPUs due to a lack of
sufficient parallelism. To run the test suite in a reasonable amount of
time, you may wish to exclude the matter-energy exchange tests, e.g.:

::

ctest -E "MatterEnergyExchange*"

which should end with output similar to the following:

::

100% tests passed, 0 tests failed out of 18

Total Test time (real) = 353.77 sec

AMD GPUs *(experimental, use at your own risk)*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Compile with ``-DAMReX_GPU_BACKEND=HIP``. Requires ROCm 5.2.0 or newer.
Your MPI library **must** support GPU-aware MPI for AMD GPUs. Quokka has
been tested on MI100 and MI250X GPUs, but there are known compiler
issues that affect the correctness of simulation results (see
https://github.com/quokka-astro/quokka/issues/394 and
https://github.com/quokka-astro/quokka/issues/447).

Intel GPUs *(experimental, use at your own risk)*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Not tested. You can attempt this by compiling with
``-DAMReX_GPU_BACKEND=SYCL``. Please start a Discussion if you encounter
issues on Intel GPUs. Your MPI library **must** support GPU-aware MPI
for Intel GPUs.

Building a specific test problem
--------------------------------

By default, all available test problems will be compiled. If you only
want to build a specific problem, you can list all of the available
CMake targets:

::

cmake --build . --target help

and then build the problem of interest:

cmake .. -DCMAKE_BUILD_TYPE=Release -DAMReX_GPU_BACKEND=CUDA
make -j6
::

The compiled test problems are in the test problem subdirectories in `build/src/`. Example scripts for running Quokka on compute clusters are in the `scripts/` subdirectory. Please note that you must configure your compute cluster to run with 1 MPI rank per GPU in order for Quokka to work correctly. Quokka is only supported on Volta-class (V100) GPUs or newer.
ninja -j6 test_hydro3d_blast

**AMD or Intel GPUs:** Running on AMD or Intel GPUs is currently experimental and has *not been tested* by the Quokka developers. AMReX is currently undergoing rapid advances in its support for GPUs from these vendors, so please get in touch by starting a Discussion before attempting this.

0 comments on commit f4784d2

Please sign in to comment.