Update submodules, API changes, tracer swarm packs, Monte Carlo packs, formatting CI

[AstroBarker]

PR Summary

Far overdue, this PR updates parthenon, singularity-eos, and singularity-opac and makes necessary changes to work with their updated versions. There are a number of additional changes that impact performance.

• Updates to MHD (API changes for packing)

• Overhauls to swarm API

• The advection problem's tracers were refactored due to a bug on multiple blocks

• Some updates to BCs, necessitated by parthenon changes. Probably more simplification can be done.

  • Need to use gr_outflow instead of outflow as the latter is now a Parthenon builtin that isn't spacetime / geometry aware.

• The fast logs of singularity-eos are default disabled.

• Updates neutrino mean opacity API

• Tracers:

  • Use SwarmPacks
  • Refactored to work on MeshData instead of MeshBlockData
  • tracer swarm variables are SWARM_VARIABLE now, so access with e.g., tv::rho(), tv::rho::name() e.g., (for namespace tv = tracer_variables; to enable type based packing.

• Started porting moments code to packs

• Monte carlo code was ported to swarm and sparse packs. Soon: to MeshData

• Monte Carlo swarm variables are VARIABLE, SWARM_VARIABLE or TENSOR_VARIABLE.

• MOCMC variables are SWARM_VARIABLE and TENSOR_SWARM.

• A consequence of the packing and template variable names is that the neutrino species is now compile time. It defaults to all 3 flavors enabled and is controlled as -DPHOEBUS_DO_NU_ELECTRON=On, -DPHOEBUS_DO_NU_ELECTRON_ANTI=On, and -DPHOEBUS_DO_NU_HEAVY=On

• I have implemented MeshData level swarm comms routines (from Jaybenne)

• I have implemented a temporary MeshData level Monte Carlo MomentFluidSource routine that for loops over blocks.

  • Later this should be implemented to other swarms machinery

• Also, the github ubuntu-latest changed and clang-format-12 was no longer available. I have moved us to installing clang-format with pip, pinned at version 12.0.1. Which was not compatible with ubuntu's v12...

• Weirdness seems to have crept in to the regression testing. Some tests, despite producing an identical gold file (when ran with --upgold) failed tests due to how hdf5 was loaded. Unsure what caused this change, but a fix was implemented.

[Yurlungur]

This was a BIG effort. Thanks @AstroBarker . Some minor nitpicks that should be addressed before merging. But I'm approving now. Nice work!

[Yurlungur]

I wonder if the specialization isn't working properly. Maybe worth double checking that this version of the code is actually being called. (And same for the others...)

[Yurlungur]

Does GetBlockData(b) definitely return the right stage? it should I think? Might be worth double checking the name that gets spit out here

[bprather]

Ah! This is straight from a MeshData object -- that provides the correct stage, yeah.

pmb->meshblock_data.Get(); does not -- I was sensitive because we just got bit by this in KHARMA: parthenon-hpc-lab/parthenon#1220

Personally I like the foreach pattern Philipp suggests in that issue over numerical iteration.

[Yurlungur]

Since b3d4ebf isn't working lets just revert for now. Can be a fix for a later MR.

[carlnotsagan]

is there a reason these BC flags are preserved? is this a cartesian coords thing