Home

Code structure

SPH-EXA is a Smoothed Particle Hydrodynamics code in 3D. It implements a few physical scenarios, referred below as test cases.

Code is structured as follows:

SPH-EXA
|   README.md
└───include/        - folder containing all sph functions,
|                     utilities and helpers that are shared between all test cases.
└───src/            - folder containing test cases
    └───testcase/   - folder containing test case main function and 
                      other test case specific functions, 
                      like reading input data from the file

Each test case is compiled separately and therefore there is the main function for each test case.

The main test case file, i.e. src/sqpatch/sqpatch.cpp can be divided into the following steps.

1. Reading command line arguments

2. Reading from file or generating input for a test case

   As a result of this step, object of the ParticleData class is created. That object holds all particles data and it's served as an input and output in all sph and domain distribution functions

3. Running a loop for s iterations (time-steps)

   An essential part of each sph test case. It can be further divided into the following steps:

   1. Domain distribution
   2. Map particles in 3D to an Octree (buildTree)
   3. Find neighbors of each particle
   4. Call sph functions calculating physics i.e. computeDensity, computeMomentumAndEnergy functions
   5. Update positions of all particles
   6. (Optionally) Dump particle data to file

Following sequence diagram illustrates the flow for the sample test case that doesn't need gravity calculations i.e. Square Patch:

SPH functions

Each function takes two parameters: TaskList and Dataset objects. TaskList is a list of Task objects, which is used to determine which particle indexes should be calculated in the particular function call. Dataset contains data for all particles.

Octree and domain distribution

3D space is mapped to an Octree in SPH-EXA. The octree is used to perform some sph calculations (such as sph::findNeighbors) as well as to do a domain distribution across MPI ranks.

Octree is divided into two parts, the global and local ones. The global part of the Octree is known to all MPI ranks in terms of structure, but not in terms of data. This tree is used in domain distribution to assign MPI ranks to the tree nodes. When MPI ranks are assigned, the data exchange happens. If rank was assigned to the particular tree node, it will receive all particle data belonging to that node. Then the given node can build the local tree from this data. If the node was not entirely assigned to the MPI rank, the assignee value will be equal to -1. It's an indication that this tree node is shared between at least two MPI ranks. This assumption is later used in calculating gravity in the distributed manner, see the description below.

Global tree nodes that are shared between more than one MPI rank (Octree::assignee equals to -1)

It's important to calculate gravity contributions from the shared tree nodes only once. Hence remoteGravityTreeWalks function will skip calculating contribution from the tree nodes that have assignee equal to -1, because those were calculated during the self-gravity calculations. This assumption is also used in the build global gravity tree phase (sph::buildGlobalGravityTree). It's needed to synchronize gravity parameters only in the tree nodes that are shared. Only this and local tree nodes will be used to calculate gravity in the self-gravity calculation phase. Hence the function sph::buildGlobalGravityTree has to be called before sph::gravityTreeWalk for MPI runs. If given particle needs contribution from the tree node that is assigned to other MPI rank, that particle will be sent to that MPI rank with a request to calculate contribution there.

Gravity calculations

Gravity is calculated using the octree data structure (GravityOctree class). Each particle is traversing the tree recursively and calculates the gravity contribution of the tree nodes (sph::gravityTreeWalk).

To perform gravity calculations, the Domain object must be created with the template parameter of GravityOctree. Thanks to that, the Octree will be built with additional parameters per node, which will be used later to calculate gravity contributions in the sph::gravityTreeWalk function.

To calculate gravity in the test case, the following steps are needed: 0. ensure that you build the code with -DGRAVITY compile flag

1. pass a GravityOctree class as a template parameter to Domain class Then for every iteration (time-step) in the test case:
2. In the main iteration loop: call function sph::gravityTreewalk

The steps above will only work for non-MPI versions of the code. To be able to calculate gravity in a distributed manner, additional steps are needed.

Distributed gravity calculations

The Octree in SPH-EXA is divided into two parts, the global and the local ones. The global part, referred as a Global Octree or Global Gravity Octree is known to all computing nodes (MPI ranks) in terms of structure. See section explaining these two trees in details

Every compute node has data of only its particles and its neighbors. Gravity is a global force, therefore one node will need to calculate the gravity contribution not only from data it contains, but from some other nodes, or even all other nodes in the worst case.

The first step is to build the Global Gravity tree (function sph::buildGlobalGravityTree). This step is needed to synchronize the gravity parameters of the global part of the tree with all the nodes. Every node needs to have the same gravity parameters in the tree nodes, to be able to correctly distinguish if a particle needs gravity contribution from the other node.

If the particle during calculations in the sph::gravityTreeWalk function needs information from the other node to perform calculations, this particle is marked as a particle to send to that node, to perform gravityTreewalk there again and gather back the needed data. Hence the gravityTreeWalk calculations are divided into two parts, gravityTreewalk for particles that are in the node (self Gravity) and gravityTreewalk for particles that other nodes request (remote or foreign particles gravityTreeWalk: sph::remoteGravityTreewalks).

The sph::gravityTreeWalk function returns a rank to particle indexes map. Each entry in the map contains particle indexes that are needed to send to the rank, to be able to get the gravity contribution from this particular rank. This map is then used as a parameter to sph::remoteGravityTreeWalks function, where the data exchange happens and additional treewalks for foreign particles are performed.

To perform gravity calculations with MPI, two additional steps are needed. Building Global Gravity Tree and performing treewalk for remote particles.

0. ensure that you build the code with -DGRAVITY compile flag
1. pass a GravityOctree class as a template parameter to Domain class Then for every iteration (time-step) in the test case:
2. Build GravityOctree (sph::buildGlobalGravityTree)
3. Perform self gravity treewalk (sph::gravityTreeWalk)
4. Perform treewalk for remote particles needing data from a given node (sph::remoteGravityTreeWalks)

Following sequence diagram illustrates the flow for test case with gravity calculations i.e. Evrard Collapse:

Adding a new test case

1. Add a new folder in path src/. The folder name must be a test case name. This name will be used later to determine the test case in the build phase

2. If the test uses gravity, compile with -DGRAVITY flag or add its name to Makefile to line stating :

        ifeq ($(TESTCASE),evrard)
	        TESTCASE_FLAGS = -DGRAVITY
        endif

3. Implement generating or reading input data from the file as well as writing output following the IFileReader and IFileWriter interfaces.
4. Create a main function in file in path src/your_testcase/your_testcase.cpp and implement test scenario