Framework: Switch CUDA AT2 build to be non-UVM and enable tests

[sebrowne]

@trilinos/framework

Motivation

Want to align the CUDA AT2 build with the old AutoTester one.

Related Issues

https://sems-atlassian-son.sandia.gov/jira/browse/TRILFRAME-673

[sebrowne]

The CUDA tests look good, with four exceptions, detailed here: https://sems-cdash-son.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=211376

@trilinos/intrepid2 I show that failing test was set to RUN SERIAL for CUDA builds, I can do that here as well if that's still what we want to do.
@trilinos/panzer that test is on for all other configs, no obvious framework-side issues to me.
@trilinos/rol that test is on for all other configs, no obvious framework-side issues to me.
@trilinos/stratimikos we had that test disabled for our non-CUDA container as well, but again, nothing really obvious from our side.

If any developers from the tagged teams can provide any insight for the four failing tests (and they do fail reliably), it would be much appreciated! I can turn them off, but I wanted to at least do SOME due diligence and see what the community thinks.

[CamelliaDPG]

@trilinos/intrepid2 I show that failing test was set to RUN SERIAL for CUDA builds, I can do that here as well if that's still what we want to do.

Yes, please. The MonolithicExecutable test is one that has a lot of test cases, and some of them are intensive, so that sharing compute resources with other tests can lead to timeouts. We use RUN SERIAL to mitigate.

[rppawlo]

@cgcgcg - would you mind taking a look at the panzer/mini-em failure here? Looks to be a linear solver issue similar to what you have fixed in the past.

[cgcgcg]

I see this message in the output of the failing Stratimikos and Panzer tests:

--------------------------------------------------------------------------
The call to cuIpcGetMemHandle failed. This means the GPU RDMA protocol
cannot be used.
  cuIpcGetMemHandle return value:   1
  address: 0x42b363c80
Check the cuda.h file for what the return value means. Perhaps a reboot
of the node will clear the problem.
--------------------------------------------------------------------------

[masterleinad]

I see this message in the output of the failing Stratimikos and Panzer tests:

That looks like issues with cudaMallocAsync, see https://kokkos.org/kokkos-core-wiki/known-issues.html?highlight=known+issues#cuda, and https://kokkosteam.slack.com/archives/C5BGU5NDQ/p1726216998539829.

[cgcgcg]

@sebrowne Do we set Kokkos_ENABLE_IMPL_CUDA_MALLOC_ASYNC=OFF for our Cuda builds?

[sebrowne]

We do not. I did do some debuggery and that particular error went away when I disabled the smcuda btl in OpenMPI. I can try that option as well (the test I was using to debug did still fail my way, but it was a NaN in Belos instead of the CUDA traceback).

[sebrowne]

New results with the Kokkos option, my disable of the smcuda BTL, and running the Intrepid2 test serially: https://sems-cdash-son.sandia.gov/cdash/viewTest.php?onlyfailed&buildid=217579

Seeing the same tests fail (except for the Intrepid2 one), but in perhaps more-straightforwards way? I see NaN errors from Belos.

[cgcgcg]

@sebrowne Thanks for adding the option. Seems like the message went away. I'll have another look to see what's wrong.

[jhux2]

Thanks for adding the option. Seems like the message went away. I'll have another look to see what's wrong.

Fyi, The option -DKokkos_ENABLE_IMPL_CUDA_MALLOC_ASYNC=OFF resolved the same warning I was seeing on the CEE lan, and more importantly it let to a 2x speedup.

[masterleinad]

We decided to make -DKokkos_ENABLE_IMPL_CUDA_MALLOC_ASYNC=OFF the default again in Kokkos (but that will likely only be visible in Trilinos with the next release).

[cgcgcg]

I just went down the rabbit hole. The issue in this test
https://sems-cdash-son.sandia.gov/cdash/test/8036828
is caused by a bug in OpenBLAS before 0.3.27
OpenMathLib/OpenBLAS@7e9b1c0
(Essentially, a double variable was wrongly interpreted as complex double and then the imaginary part was used in a division. If the imaginary part happened to be zero this resulted in a NaN.)
Updating the container to a more recent version of OpenBLAS should fix that test.

[sebrowne]

Perfect, thank you so much! I'll get on fixing it ASAP.