Quokka 24.04

We are excited to announce Quokka 24.04, the second release of Quokka (tagged with the calendar versioning [year.month]). Quokka continues to evolve, addressing the needs of researchers in the field.

This release has been well-tested across platforms, ensuring reliable performance on CPUs (both Linux and macOS) and GPUs from NVIDIA, with experimental support for AMD GPUs (see the documentation for details). An automated test suite of 1D, 2D, and 3D problems is provided in the src subdirectory.

Release managers for this release: @chongchonghe @BenWibking

What's Changed

1. Dual energy formalism for hydro
2. Hydro robustness:
   • First-order flux correction triggered for bad cells
   • Timestep retries triggered by bad cells
   • Better wavespeed estimates for the HLLC Riemann solver
3. Self-gravity (but without AMR subcycling)
4. Navier-Stokes Characteristic Boundary Conditions for subsonic inflows/outflows
5. Support for openPMD/ADIOS2 outputs
6. Support for in-situ diagnostics:
   • N-D Histograms
   • 2D Slices
   • 2D Projections
   • History file output
7. Tracer particles
8. Collisionless gravitating particles (CICParticles)
9. Updated schemes for the radiation subsystem
10. IMEX PD-ARS Scheme Implementation. The radiation subsystem and matter-radiation coupling source terms now utilize the newly implemented IMEX PD-ARS scheme for time integration, replacing the original SSP-RK2 scheme.
11. Radiation fluxes via unmodified Riemann solver. The ad hoc modifications to the wavespeed computed in the Riemann solver is no longer necessary given the IMEX scheme's asymptotic-preserving property.
12. A fully implicit update of the matter-radiation coupling source terms that includes up to $O(v^2/c^2)$ terms in the radiation four-force.
13. Preliminary formulation of multigroup radiation hydrodynamics: The user can define arbitrary group edges and specify Planck-mean, flux-mean, and energy-mean opacities for each photon group as a function of gas density and temperature. This is a preliminary implementation of multigroup RHD. In our next release we will include variable opacity interpolation and multigroup radiation advection.
14. Tabulated cooling
15. Integration with amrex-astro/Microphysics for equation-of-state and reaction networks
    • Primordial chemistry
    • General (convex) EOS support

Changelog

(Submodule version updates are omitted from this list.)

• implement carbuncle-free low-dissipation HLLC solver by @BenWibking in #18
• implement first-order flux correction (FOFC) for hydro by @BenWibking in #25
• add optically-thin cooling (radiative, collisional, etc.) by @BenWibking in #24
• Robustness fixes for cooling + AMR by @BenWibking in #28
• More cooling solver fixes by @BenWibking in #30
• Enable regridding on AMR subcycles by @BenWibking in #31
• Enable Debug + ASAN in CI by @BenWibking in #32
• Add performance hints by @BenWibking in #39
• add automated test for carbuncle instability by @BenWibking in #40
• inline ComputeVelocityDifferences into ComputeFluxes by @conradtchan in #35
• add energy checks to Sedov test by @BenWibking in #41
• Fuse PPM reconstruction kernels by @conradtchan in #36
• pause simulation to flush async outputs if needed by @BenWibking in #43
• add support for Chombo-format HDF5 plotfile outputs by @BenWibking in #44
• allow building device code with Clang by @BenWibking in #42
• add support for Ascent by @BenWibking in #29
• add max_walltime parameter by @BenWibking in #46
• fix carbuncle correction in 2D by @BenWibking in #48
• add Sedov verification scripts and notebook by @BenWibking in #49
• add stop_time, plotfile_interval, and checkpoint_interval parameters by @BenWibking in #51
• add CFL-respecting AMR subcycling by @BenWibking in #52
• add CONTRIBUTING.md based on AMReX's version by @BenWibking in #54
• add auxiliary internal energy equation for high-Mach flows by @BenWibking in #27
• include ghost cells in plotfiles and ascent renders by @BenWibking in #55
• allow arbitrary number of scalars in riemann solver by @BenWibking in #56
• rename EOS_Traits to HydroSystem_Traits by @BenWibking in #60
• add passive scalars by @BenWibking in #61
• add passive scalar test problem by @BenWibking in #62
• adjust dual energy 'eta' parameter by @BenWibking in #59
• avoid allocating radiation variables for hydro-only problems by @BenWibking in #64
• add ascent_interval runtime parameter by @BenWibking in #65
• use fast scaled interpolation of cooling tables by @BenWibking in #66
• arm64 build fix by @BenWibking in #72
• make functions accessed in initializer list static by @BenWibking in #77
• fix initial conditions in HydroQuirk by @BenWibking in #76
• change the way CMake finds Python headers by @BenWibking in #79
• add reconstruction order input params by @BenWibking in #80
• fix cmake python support by @BenWibking in #82
• add use_dual_energy parameter by @BenWibking in #81
• run CI on pull requests by @BenWibking in #84
• Disable Sonarcloud analysis on external PRs by @BenWibking in #85
• initialise auxiliary gas internal energy in test problems by @AstroKriel in #83
• move sonarcloud analysis to separate CI action by @BenWibking in #89
• add Physics_Traits compile-time options by @AstroKriel in #88
• specify max_timesteps at runtime for HydroBlast3d by @BenWibking in #90
• add AMREX_GPUS_PER_NODE to Gadi host config by @BenWibking in #91
• add Strang-split sources to hydro update by @BenWibking in #94
• Update PR guidelines by @BenWibking in #97
• Added include guards for the physics-trait header files by @AstroKriel in #99
• Fix periodicity bug in the rad-hydro-shell simulation by @AstroKriel in #98
• compute internalEnergy P dV with face-centered velocities by @BenWibking in #95
• reconstruct SIE for internal energy variable by @BenWibking in #100
• Fix P -> Pgas in Hydro_System assert by @BenWibking in #103
• add CI debug build on arm64 by @BenWibking in #105
• Fix bug in P dV term in dual energy update by @BenWibking in #107
• Refactor simulation initialisation by @AstroKriel in #106
• Remove dimension dependent (3D) import of ArrayView by @AstroKriel in #108
• add Rayleigh-Taylor test problems by @BenWibking in #110
• add image of RT3D simulation to README.md by @BenWibking in #111
• fix shock-cloud initialization by @BenWibking in #114
• use internal energy for AMR interpolation by @BenWibking in #113
• compute 1D averages of user-specified quantities by @BenWibking in #115
• added CMake config for NCSA Delta by @BenWibking in #116
• Update NCSA Delta config by @BenWibking in #119
• use MultiFab ParallelFor for RT2D/3D problems by @BenWibking in #120
• enable gpu-aware MPI by default by @BenWibking in #121
• update MI100 build settings by @BenWibking in #122
• update amrex submodule by @BenWibking in #123
• Rename variables to indicate they store cell-centred quantities by @AstroKriel in #126
• add CMake settings for Crusher (OLCF) by @BenWibking in #127
• disable pltfile output for shocktube test by @BenWibking in #128
• add benchmark parameter files for Crusher by @BenWibking in #130
• apply density floor after regrid by @BenWibking in #132
• avoid filling old_state when regridding by @BenWibking in #131
• allow limiting timesteps based on extra physics by @BenWibking in #134
• fix undefined behavior in AddInternalEnergyPdV by @BenWibking in #136
• suppress gcc warnings about ARM ABI change by @BenWibking in #137
• fix undefined behavior in ComputeRhsFromFluxes by @BenWibking in #138
• fix memory errors in setInitialConditionsOnGrid by @BenWibking in #139
• run CI with ENABLE_ASAN=ON by @BenWibking in #141
• fix memory issue in ShockCloud initial conditions by @BenWibking in #140
• update GitHub actions by @BenWibking in #142
• Update input argument to user initialisation function by @AstroKriel in #129
• remove dt assert in ODEIntegrate by @BenWibking in #143
• decide whether to subcycle based on cost by @BenWibking in #145
• use MultiFab ParallelFor for hydro by @BenWibking in #135
• Rename grid member variables by @AstroKriel in #147
• Rename BCs-variable to indicate centering by @AstroKriel in #146
• add ability to retry hydro advance with smaller dt by @BenWibking in #148
• use FillPatcher class for ghost cell filling when using AMR by @BenWibking in #133
• retry hydro step on post-hoc CFL violation by @BenWibking in #149
• fix reflux when timestep retries are used by @BenWibking in #150
• Fix bug in additional subcycling by @BenWibking in #151
• disable additional subcycling code by @BenWibking in #154
• add refinement to passive scalar test by @BenWibking in #155
• use CellConservativeProtected in 2D and 3D by @BenWibking in #156
• use lincc_interp AMR interpolation method by default by @BenWibking in #157
• update settings for Crusher (OLCF) by @BenWibking in #159
• update amrex submodule by @BenWibking in #161
• use custom AMR interpolation method by @BenWibking in #163
• fix bug in first-order flux correction by @BenWibking in #164
• use GPU reduction for checkHydroStates by @BenWibking in #166
• require CUDA 11.7+ by @BenWibking in #168
• update .clang-format by @BenWibking in #169
• run clang-tidy on PRs by @BenWibking in #170
• use .clang-tidy file with clang-tidy-review by @BenWibking in #171
• have clang-tidy read compile_commands.json by @BenWibking in #172
• fix clang-tidy-review GitHub action by @BenWibking in #173
• update .clang-tidy config file by @BenWibking in #174
• ignore clang-tidy array index warning by @BenWibking in #175
• Update README.md with CUDA version requirement by @BenWibking in #178
• Add + turn off MHD flag (MHD is not implemented yet) by @AstroKriel in #177
• use SimulationData for problem-specific data by @BenWibking in #180
• use the amrex version of the new interpolater by @BenWibking in #181
• reduce memory usage in hydro retries by @BenWibking in #182
• use adaptive wavespeed estimates for Riemann solver by @BenWibking in #183
• fix cooling failure at high densities by @BenWibking in #184
• refactor hydro riemann solver by @BenWibking in #185
• self-gravity for non-subcycling-in-time AMR by @BenWibking in #167
• remove unused MultiFabs in RemakeLevel by @BenWibking in #187
• update build instructions in README by @BenWibking in #188
• Add ability to create and work with face-centred quantities by @AstroKriel in #125
• Add more notes about GPU-aware MPI to README by @BenWibking in #189
• refactor hydro Riemann solver in canonical form by @BenWibking in #190
• enforce code formatting by @BenWibking in #191
• Read cc and fc ncomp from Physics_Indices by @AstroKriel in #197
• create EOS and EOS_Traits classes by @BenWibking in #196
• Add HLLD solver for ideal MHD by @AstroKriel in #195
• fix build error with CUDA 12 by @BenWibking in #200
• Enforcing upper/lower limits to density, temperature and speed. by @aditivijayan in #192
• disable readability-function-cognitive-complexity warning by @BenWibking in #201
• avoid clang-tidy-review post on merged PRs by @BenWibking in #208
• create GitHub Action for docs by @BenWibking in #205
• Changed box coordinates for the SphericalCollapse test problem by @psharda in #206
• add flowchart to docs by @BenWibking in #203
• fix publish docs action by @BenWibking in #211
• avoid using inf for limits by @BenWibking in #210
• fix branch name in docs.yml by @BenWibking in #212
• update amrex submodule by @BenWibking in #213
• Increased pressure for spherical collapse gravity test by @psharda in #215
• update SphericalCollapse parameters by @BenWibking in #219
• use Roe-average wavespeeds for HLLC by @BenWibking in #217
• allow simulation to continue if retries exceed max by @BenWibking in #209
• Timed output by @aditivijayan in #227
• workaround constexpr error on Clang-based compilers by @BenWibking in #222
• fix AppleClang build by @BenWibking in #229
• add multidimensional artificial viscosity by @BenWibking in #228
• update SLURM script for Crusher by @BenWibking in #226
• enable cooling with a runtime parameter by @BenWibking in #225
• adjust multigrid parameters for gravity solve by @BenWibking in #223
• update amrex to fix macOS build by @BenWibking in #232
• add GitHub CI for macOS by @BenWibking in #233
• update amrex submodule by @BenWibking in #234
• Fix typos in the HLLD implementation by @AstroKriel in #236
• update Delta (NCSA) config by @BenWibking in #235
• added Microphysics as a submodule for Quokka by @psharda in #216
• Updating cmakelists in quokka to include microphysics by @psharda in #240
• Add ability to specify which quantities to PPM reconstruct by @AstroKriel in #244
• update amrex submodule by @BenWibking in #246
• avoid deprecated Python API call by @BenWibking in #247
• Disable parallel builds in CI by @BenWibking in #248
• increase azure-pipelines CI time limit by @BenWibking in #249
• remove CodeQL CI by @BenWibking in #250
• Create dependabot.yml by @BenWibking in #254
• fix clang-tidy-review from forks by @BenWibking in #255
• Update sonarcloud.yml by @BenWibking in #264
• Use macOS 13 on GitHub Actions by @BenWibking in #266
• cancel in-progress CI builds on new push by @BenWibking in #268
• Patched microphysics to quokka, and added primordial chem test by @psharda in #263
• check spelling in source code by @BenWibking in #277
• Added mass scalars and consistent multi-fluid advection (CMA) by @psharda in #273
• added Setonix (GPU) build scripts by @BenWibking in #282
• update Setonix SLURM script by @BenWibking in #284
• Remove ReadTheDocs badge from README by @BenWibking in #288
• optimise load by @aditivijayan in #287
• treat all compiler warnings as errors by @BenWibking in #283
• simplify Setonix SLURM scripts by @BenWibking in #289
• create new GitHub action for compiler warnings by @BenWibking in #293
• Add massScalars as an optional argument to EOS routines by @psharda in #291
• Using CMake object library to compile files needed by all tests by @psharda in #295
• Compute massScalars and use them in EOS routines by @psharda in #296
• include massScalars in call to EOS functions in hydro and radiation by @psharda in #298
• Ensure EOS.hpp compiles on AMD GPUs by @psharda in #310
• use two cores to run arm64 debug pipeline by @psharda in #317
• update Setonix CMake settings by @BenWibking in #321
• Update dependencies_hip.sh to use ROCm 5.4.3 by @BenWibking in #318
• Two EOS compilation for Quokka + fix build on cray systems by @psharda in #311
• updated job scripts for Setonix by @BenWibking in #331
• Use microphysics EOS in Quokka's EOS.hpp for all quokka tests + use amu instead of hydrogen mass throughout Quokka by @psharda in #324
• Compute pressure from microphysics eos by @psharda in #335
• Implement general EOS equations in Riemann Solver by @psharda in #338
• Starcluster example problem by @psharda in #342
• Update microphysics by @psharda in #351
• add Navier-Stokes Characteristic Boundary Conditions by @BenWibking in #352
• add yaml-cpp as a submodule by @BenWibking in #360
• workaround Ascent issue by @BenWibking in #365
• fill ghost cells before output by @BenWibking in #363
• add history file output by @BenWibking in #366
• Remove is_chemistry_enabled by @psharda in #369
• add axis-aligned projection outputs by @BenWibking in #367
• add YAML metadata I/O by @BenWibking in #364
• update setonix configuration by @BenWibking in #362
• Bump actions/checkout from 3 to 4 by @dependabot in #370
• revert to first-order space+time flux when RK2 flux fails by @BenWibking in #221
• ensure first-order flux correction is consistent at box boundaries by @BenWibking in #376
• add LLF fallback solver by @BenWibking in #380
• Enforce checks and limits on mass scalars by @psharda in #378
• output multifab to disk if hydro update fails by @BenWibking in #385
• update amrex by @BenWibking in #387
• use minmod limiting for PLM by @BenWibking in #384
• Print coordinates of cells where flux correction is needed by @psharda in #383
• update the flowchart on the website by @chongchonghe in #392
• Valarray extended by @chongchonghe in #391
• Update README by @psharda in #398
• Add sample Setonix scripts by @psharda in #399
• Fix indentations by @psharda in #407
• Create issue templates by @BenWibking in #414
• Fix typos found by codespell by @BenWibking in #416
• Update GPU CI timeout by @BenWibking in #417
• Multigroup radiation transport implementation by @chongchonghe in #388
• Renormalize mass scalars post enforcing floor by @psharda in #405
• workaround macOS GitHub actions bug by @BenWibking in #425
• fix out-of-bounds bug in NSCBC outflow by @BenWibking in #430
• add Summit (OLCF) config by @BenWibking in #435
• fix src/CMakeLists for macOS with Clang by @chongchonghe in #395
• add CI pipeline for AMDGPU by @BenWibking in #442
• Add build configuration for Moth by @BenWibking in #440
• Fix GPU CI config by @BenWibking in #443
• add build profile for AMDGPU ASAN by @BenWibking in #446
• fix odd-even decoupling in radiation diffusion limit by @BenWibking in #453
• Extend HydroState to include MHD-vars for HLLD solver by @AstroKriel in #420
• Fix the convergence problem of the Newton-Raphson by @chongchonghe in #454
• adding parameters.rst to docs by @aditivijayan in #432
• Add pull_request_template.md by @BenWibking in #452
• move analysis pages from GitHub wiki to Sphinx docs by @BenWibking in #459
• move cluster-specific instructions from wiki to Sphinx docs by @BenWibking in #461
• Use IMEX PD-ARS scheme for radiation update by @chongchonghe in #448
• add chapter headings and developer guide to Sphinx docs by @BenWibking in #465
• move numerical stability page from wiki to Sphinx docs by @BenWibking in #460
• add support for writing openPMD/ADIOS2 plotfiles by @BenWibking in #466
• Fallback to zero-gradient extrapolation when NSCBC fails by @BenWibking in #472
• use frexp and ldexp in FastMath by @BenWibking in #471
• update moth profile for ROCm 6.0 by @BenWibking in #475
• add additional metadata to openPMD outputs by @BenWibking in #473
• radiation advection by @chongchonghe in #462
• Multigroup radiative shock test by @chongchonghe in #467
• add tests to sphinx docs by @chongchonghe in #483
• Define advecting pulse test problem. by @chongchonghe in #463
• document hydro, rad, and cooling runtime parameters by @BenWibking in #488
• fix macOS CI by @BenWibking in #496
• Use correct units for mean molecular weight by @psharda in #494
• Save gravitational potential gpot as a derived var in pltfiles by @psharda in #406
• add tracer particles by @BenWibking in #490
• build GitHub actions in parallel by @BenWibking in #501
• add CIC particles by @BenWibking in #502
• Make ComputeRadPressure return F and S by @chongchonghe in #512
• update macOS CI to use M1 runner by @BenWibking in #513
• add HWASAN CI run by @BenWibking in #510
• re-enable CodeQL by @BenWibking in #514
• Multigroup advecting pulse test by @chongchonghe in #486
• Fix RadShadow test problem by @BenWibking in #476
• change projection plotfile names by @BenWibking in #523
• PopIII Simulation by @psharda in #368
• Fix bug in perturbation.py by @psharda in #525
• fix copies to rank zero for correctness tests by @BenWibking in #531
• fix clang-tidy warnings by @BenWibking in #468
• Newton iteration on new basis (Egas, R / tau0) by @chongchonghe in #535
• Define the grey-radiation advecting pulse test by @chongchonghe in #538
• fix final checkpoint file write by @BenWibking in #558
• Defined advecting pulse test in dynamic diffusion regime by @chongchonghe in #551
• compile warnings with AMREX_SPACEDIM=3 by @BenWibking in #564
• Fix clang-tidy-review comment posting by @BenWibking in #567
• run CodeQL analysis with AMREX_SPACEDIM=3 by @BenWibking in #568
• output 2D slice plotfiles by @BenWibking in #562
• Update Factory.H to use std::move by @BenWibking in #576
• Define K_S for isothermal gamma by @psharda in #573
• Avoid imaginary roe average sound speed by @psharda in #572
• Fixed bug in PopIII refinement by @psharda in #574
• New source terms that include the work term by @chongchonghe in #552
• Fix amrex::abort not aborting unconditionally on GPUs (for primordial chem and popiii problems) by @psharda in #575
• Remove redundant parameter compute_v_over_c_terms by @chongchonghe in #579
• fix face-centered BCs for refined levels by @BenWibking in #584
• add histogram (PDF) output by @BenWibking in #581
• update histogram plotting script by @BenWibking in #588
• docs: add in-situ analysis by @BenWibking in #586
• add simple subsonic inflow/outflow BCs by @BenWibking in #598
• fix tracer particle handling in hydro retries by @BenWibking in #596
• add computeVolumeIntegral function by @BenWibking in #597
• Remove wavespeed correction in radiation flux by @chongchonghe in #582
• Update test parameters to produce all the figures in the IMEX paper by @chongchonghe in #600
• Refactor Reconstruction Functions by @AstroKriel in #570
• add "experimental" warning for AMD and Intel GPUs by @BenWibking in #602
• Fix Newton-Rasphon convergence bug when tau = 0 by @chongchonghe in #603
• read cloudy-cooling-tools tables by @BenWibking in #599
• Move installation instructions to docs by @chongchonghe in #607
• fix typo in TabulatedCooling by @BenWibking in #606

New Contributors

• @conradtchan made their first contribution in #35
• @AstroKriel made their first contribution in #83
• @aditivijayan made their first contribution in #192
• @psharda made their first contribution in #206
• @chongchonghe made their first contribution in #392

Full Changelog: 21.10...24.04