Add gravity method super-cycling

[jobordner]

Add super-cycled gravity, and support for super-cycling other methods.

This PR adds support for super-cycling methods, while still always satisfying method-specific timestep constraints. The purpose is to improve runtime performance, which can be especially helpful when one specific method (e.g. gravity) tends to run slower than other methods. This PR implements super-cycling for the gravity method in particular, and provides helper methods to implement super-cycling for other methods.

The time-step it uses for a super-cycled method is determined using 1. a "max_supercycle" parameter defined in the Method : <method> group, and 2. the ratio of time-steps for the super-cycled method relative to the global minimum. This allows the user to control the maximum "amount" of super-cycling, while never having to worry about violating time-step constraints.

Super-cycling works by allowing the Method to have a "solve" cycle, followed by some number k of "non-solve" cycles 0 <= k < max_supercycle. During "solve" steps, the method saves some extra fields, which are used in "non-solve" cycles to approximate the fields normally computed by the method. For the gravity method in particular, these saved fields may be either the gravitational potential, or the acceleration fields, as determined by the "Method:gravity:type_supercycle" parameter.

• New features are documented with narrative docs: see doc/source/design/design-timestep.rst

• Adds tests for new features: see input/test_cosmo-dd8.in

• [ X] This is a major change or addition. Will require two reviewers.

[mabruzzo]

Suggested change

	fields. Which fields to be extrapolated is determined by the
	fields. The fields to be extrapolated are determined by the

[mabruzzo]

Suggested change

	set(__ARCH_C_OPT_FLAGS "-O3 -g -funroll-loops")
	set(CMAKE_C_FLAGS_RELEASE "${__ARCH_C_OPT_FLAGS}")
	set(CMAKE_C_FLAGS_RELWITHDEBINFO "-g ${__ARCH_C_OPT_FLAGS}")
	set(CMAKE_CXX_FLAGS_RELEASE "${__ARCH_C_OPT_FLAGS}")
	set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "-g ${__ARCH_C_OPT_FLAGS}")

I think these are now all unnecessary due to some more recent changes.

If you look further down on the page, we define ENZOE_C_FLIST_INIT and ENZOE_CXX_FLIST_INIT.

  set(ENZOE_C_FLIST_INIT "-Wall;-funroll-loops")
  set(ENZOE_CXX_FLIST_INIT "${ENZOE_C_FLIST}")

Currently, these add the -Wall and -funroll-loops flags. If you also want the -O3 flag instead of -O2, you should modify the list so that it reads -Wall;-funroll-loops;-O3.

As an aside, we shouldn't be adding -g as a symbol here. The canonical way to get debugger symbols is to use -DCMAKE_BUILD_TYPE=RelWithDebInfo when configuring the build-system.

[mabruzzo]

If you don't specify a build-path, Enzo-E currently defaults to a "Build type" of Release. But, I would be very happy to change the default build type to RelWithDebInfo so we always get debugger symbols in the default build (I thought this should be the default when we first shifted to CMake, but it was a debate I lost at the time).

[mabruzzo]

Extrapolating the density involves the Method : pm_deposit : alpha parameter, right?

If so then our approach isn't robust in the general case. When I implemented self-gravity source terms for VL+CT, there was a whole saga where I was getting bizare looking results for a standard test problem (i.e. the "stable linear Jeans wave" that is inclined with respect to the grid and for which we can perform an analytic convergence study). Ultimately, the solution was to stop extrapolating. I suspect that we would find similar issues with PPM if we ran a similar test

In that scenario, there were no particles in that case. It's plausible that if particles dominate the short-range gravitational potential that extrapolation might be more robust.

Anyway, we probably don't want to go into detail here, but maybe we add something like the following at the end of this subsection?

.. caution:

   Some care is needed when applying extrapolation methods. For certain problems, they may not produce meaningful results.

[mabruzzo]

This isn't essential, but we may want to acknowledge that subcycling also exists (e.g. in Grackle and radiative transfer). Maybe just a single sentence noting that this is used as a technique on a method-by-method basis.