Multi-GPU deconv!

[NicoKiaru]

Hi @bnorthan,

I just wanted to share a quick update: I've successfully enabled the use of multiple GPUs (and even splitting larger GPUs into multiple instances, as shown in CLIJx Faclon Heavy) for tiled deconvolution!

Along the way, I encountered a few blocking points that I wanted to highlight:

1. NonCirculant Setting and Integrated Graphics:

To use my integrated graphics card (Intel(R) Iris(R) Xe Graphics), I had to set nonCirculant to false. When nonCirculant is true, the JVM crashes at the following line:

double mean = clij2.meanOfAllPixels(inputExtended);

This issue seems to be specific to this GPU and function, and results in a hard JVM crash.

2. Wrapper Cleanup (clij2fftWrapper.cleanup();) Is Not Thread-Safe:

When running two native deconvolve calls simultaneously, one always finishes before the other. The first to finish causes the second to crash due to the cleanup call.

If I remove the cleanup, multiple instances can run simultaneously without issue.

3. Changes to CLIJ2 Instance Creation:

I had to modify how CLIJ2 instances are created, as CLIJ2.getInstance("name") closes other instances except the one in use.

Here's a preview of how it looks with three workers (1 GPU A500 split into 2 instances + 1 Intel integrated GPU):

Multi-GPU-Deconv.mp4

Side note: I didn't use the previous Falcon Heavy work by Robert because I have a simple self-contained class for pooling, which I think could be integrated into the repo.

[NicoKiaru]

Here's my multi-gpu branch:

https://github.com/NicoKiaru/clij2-fft/commits/multi-gpu/

There are multiple modifications, a major one being the introduction of a builder pattern. That simplifies a lot the creation of objects with multiple parameters, including the creation of a GPU pool.

If you want to have a quick test if you have a machine with multiple GPUs, you can run the demo InteractiveSelfContainedImgLib2CacheDeconvolve which should be totally self-contained

[bnorthan]

This looks really good. I just grabbed with it and will investigate it more over the next few days.

[NicoKiaru]

FYI I've made a commit that's using clij/clijx-faclon-heavy#3

The big question is whether Robert has time to look at the PR. Otherwise we could make a CLIJPool class here in this repo, but that's not optimal

[bnorthan]

Hi @NicoKiaru

I got your code (InteractiveSelfContainedImgLib2CacheDeconvolve.java) to run on my machine. Pretty cool that it downloads the images right from Zenodo so no need to change file names.

I can't find Faclon-heavy on either an imagej update site (checked clij and clij2) or maven... do you know if it is uploaded to maven and/or a Fiji update site? If not I guess that is something we will have to figure out with Robert.

[NicoKiaru]

I built it locally from https://github.com/NicoKiaru/clijx-faclon-heavy/tree/master

... maybe it's good if it's not on Fiji neither in maven because then we could upload our version if Robert is super busy

[bnorthan]

I built clijx-faclon-heavy locally as well and it seemed straight forward.

I have an update site called clijx-deconvolution (I can ask Mark H. to give you access to it), which isn't being used much yet. When we are ready for a release to the update site we could try to get Robert to remove all the deconvolution and fft related artifacts from clij2 update site and add them, along with clijx-faclon to clijx-deconvolution.

It will probably also be useful to ask Robert to add you to the clijx-faclon and clij2-fft github repos. After that we should be able to do releases without having to bother Robert. @haesleinhuepf any chance you could get @NicoKiaru as a member of https://github.com/clij/clijx-faclon-heavy and https://github.com/clij/clij2-fft ?