Feature Fast RHat

[mreh528]

Pull request description

Adds a new fast and low-memory RHat calculating macro. Also adds an FFT-based calculator for the AutoCorrelation function.

Changes or fixes

Added a new RHat macro. The memory and speed savings are so significant that the previous RHat calculating macro has been moved to RHat_HighMem to make the new RHat calculator the default. The old macro lives on and isn't fully replaced since it is able to calculate the Folded RHat statistic, which the new calculator is incapable of since the main time/memory saver for us is not storing the full chain in memory (which is needed for finding the median in the Folded statistic).

The FFT-based AutoCorrelation calculator has been added inside MCMCProcessor as a separate toggle apart from the original calculator. The FFT formulation ends up producing slightly different results from the original algorithm, but shares the same characteristics as the original. The benefits are that the FFT algorithm is faster than even the GPU implementation of the original calculator, but can run on a single-threaded CPU.

[github-actions]

Hi @mreh528, thank you for contributing to MaCh3! Please wait for MaCh3 developers to review your PR. If no one answers within a week, please message people from this list: https://github.com/orgs/mach3-software/teams/mach3admin . While waiting, please enjoy this

Use this action on your projects. Use jokes on issues instead.

[github-actions]

This is your first PR, thank you for contributing to MaCh3!

[mreh528]

In order to be compatible with existing test cases, I added a "Thinning" parameter designed to skip every NThin steps when loading/diagnosing a chain. This will replace the original NToys command line argument since with the new changes, this macro is capable of loading entire chains easily.

What this means is that instead of the command line argument referring to the number of toys you want to generate, it now refers to how many steps will be skipped between loads from the chain (Default should be 1, as we expect to load the full chain with no skips).

I don't see any scripts that are predicated on the original functionality, but if any exist, they need to be updated correspondingly.

[mreh528]

Here are some plots of the intermediate calculated quantities using the old method (RED) and the new method (BLUE). Any differences in the calculated values can be attributed to the RHat calculator sampling random steps in each iteration.

[mreh528]

Here are some example autocorrelation plots comparing between the original calculator and the FFT-based algorithm using the MaCh3 tutorial test chain:

The two calculations do differ somewhat substantially, but this needs to be put into context:

• The chain used in this test is only 10,000 steps, and 10,000 lags are plotted. Much of this calculation is therefore subject to noise, and in testing the noise reduces substantially in longer chains and lower lag/step ratios
• The FFT-based algorithm is based on the Fourier transformed power spectrum, and hence only gives even function results. Hence the FFT calculation is symmetric at the midpoint

[KSkwarczynski]

Thanks a lot for this :)