Setting memory region sizes

This image shows how the memory region fractions look.

Grappa divides a node's memory into a number of regions, some private to each core and some shared with other cores and nodes. Different data is stored in each type of region; all must have some space for a Grappa program to run.

The flag --locale_shared_fraction changes the balance between using DRAM for core-local data and for shared data within a node (/locale). For historical reasons it defaults to 0.5 (half core-local and half shared); if you run out of memory you should first try setting this to 0.8 or something near that. 0.9 is probably too much.

Two other memory pools are allocated within the locale shared memory:

• --shared_pool_memory_fraction is the fraction of the locale shared memory to use for the shared heap (sending heap messages)
• --global_heap_fraction is the fraction of the locale shared memory to use for global shared heap (e.g. global and symmetric arrays)

The rest of the locale shared memory is used for data allocated with functions like locale_alloc<>() and locale_new<>(), as well as other shared runtime data structures. Some amount of locale shared memory must be available for these bits of data, so --shared_pool_memory_fraction and --global_heap_fraction both default to 0.25, and may be adjusted upwards a bit. Probably 0.5 is too much for either of these. It depends on the demands of the application.

In an ideal world we would automatically manage all of these region sizes, but since Grappa is research code we haven't written that code yet.