2023.06.28 Meeting Notes

Tentative agenda

LR

fixes bugs in sparse comm (tracking buffers)
zero-buffer alloc load balancing, ready for re-review, but no major change since first round of review
talked to PM/JM about "corrections" for (faces, edges, ...) fields
- concept of "ownership" (only once cell determines the state of a given shared element)
- a cell on the finest level should have priority
- working on framework to include this concept based on "largest morton number and highest lvl) as logical location

BP

focusing on getting kharma stable on latest develop
ran scaling tests on Frontier, ran into issues with MPI transport layers. It's a known issue

PM

ran into issue with yt on Riot outputs for scalar fields that are not length 1 vectors)
will fix (see yt below)

JM

PG

fixed Kokkos 4 and updated CI
tracking down IO issues on Frontier, added more profiling regions and allow soft-disable filter

currently fails more often than usual because the node is overused (by other work)
solution for now: just re-run fail jobs and they'll eventually pass (highest likelyhood during nighttime in Germany)
PG applied for money to get a separate CI node but will only know mid/end July if that goes through
Check with OLCF if there's infrastructure we could use (Slate?)

move fast to get it upstream
focus on clean frontend for AthenaPK and a working frontend for other codes (i.e., can work with raw data etc but no guarantee for automatic derived fields etc)
other frontends can be added/expanded later when required

PM discovered round of level error in few ghost cells even at the first cycle when run on different number of ranks (CPU only runs)
error seems to show up at face where prolongation happens
suggested further debugging approach
- change so simple prolongation ( to reduce prolongation stencil)
- use different number of meshblock packs (currently just one pack per rank)
- try to compare to a GPU run (currently only CPU runs)