Test act_uniform_alm failed

[ArnoldTUB]

Hello everyone,

I installed AMR-Wind on a Ubuntu 22.04 server. After running the command ctest --output-on-failure all tests but the last one pass.
Do you have any idea what's causing this error? And how crucial is whatever's being tested by this routine for running simulations on flat terrain with not more than 10 turbines?

Best
Arnold

  AMR-Wind (https://github.com/exawind/amr-wind)
  AMR-Wind version :: v3.4.0-2-gc6a54a71-DIRTY
  AMR-Wind Git SHA :: c6a54a719af70a0095e82e47c0bbf2618aafdfeb-DIRTY
  AMReX version    :: 25.02-10-g63e03ea81732
  Exec. time       :: Thu Feb 13 17:23:25 2025
  Build time       :: Feb 13 2025 15:40:22
  C++ compiler     :: GNU 11.4.0
  MPI              :: ON    (Num. ranks = 4)
  GPU              :: OFF
  OpenMP           :: ON    (max threads = 2)
  Enabled third-party libraries: 
    OpenFAST  
           This software is released under the BSD 3-clause license.           
 See https://github.com/Exawind/amr-wind/blob/development/LICENSE for details. 
------------------------------------------------------------------------------
[ab2eea78d0ac:02245] Read -1, expected 9326, errno = 1
[ab2eea78d0ac:02246] Read -1, expected 9326, errno = 1
[ab2eea78d0ac:02247] Read -1, expected 9326, errno = 1
Initializing AMReX (25.02-10-g63e03ea81732)...
MPI initialized with 4 MPI processes
MPI initialized with thread support level 0
OMP initialized with 2 OMP threads
AMReX (25.02-10-g63e03ea81732) initialized
Initializing AMR-Wind ...
Initializing AMR-Wind.
Initializing boundary conditions for velocity_mueff
Initializing boundary conditions for velocity
Initializing boundary conditions for density
Initializing boundary conditions for p
Initializing boundary conditions for velocity_src_term
Initializing boundary conditions for gp
Creating PDESystem instance: ICNS-Godunov
Creating TransportModel instance: ConstTransport
Creating Physics instance: FreeStream
Creating Physics instance: Actuator
Creating TurbulenceModel instance: Laminar-ConstTransport
Creating RefinementCriteria instance: GeometryRefinement
Creating GeometryType instance: box
Creating RefinementCriteria instance: GeometryRefinement
Creating GeometryType instance: box
Creating RefinementCriteria instance: GeometryRefinement
Creating GeometryType instance: box
Creating RefinementCriteria instance: GeometryRefinement
Creating GeometryType instance: box
Creating RefinementCriteria instance: GeometryRefinement
Creating GeometryType instance: box
Creating RefinementCriteria instance: GeometryRefinement
Creating GeometryType instance: box
Creating RefinementCriteria instance: GeometryRefinement
Creating GeometryType instance: box
Creating RefinementCriteria instance: GeometryRefinement
Creating GeometryType instance: box
Initializing I/O manager
Creating UDF instance: ConstValue
Creating UDF instance: ConstValue
Creating ActuatorModel instance: TurbineFastLine
Initializing turbine:T0
TurbineFast: no density conflict detected.
Creating ActuatorModel instance: TurbineFastLine
Initializing turbine:T1
TurbineFast: no density conflict detected.
Creating mesh... done
Grid summary: 
  Level 0   27 grids  884736 cells  100 % of domain
            smallest grid: 32 x 32 x 32  biggest grid: 32 x 32 x 32
Creating ExtSolver instance: OpenFAST
Erroneous arithmetic operation
See Backtrace.0.0 file for details
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode 8.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

[marchdf]

I am not sure exactly what is going on here. But we've had issue with OpenMP and openfast. So maybe try a non-openmp build?

[ArnoldTUB]

I retried with MPICH and receive a similar error:

Creating ExtSolver instance: OpenFAST
Erroneous arithmetic operation
See Backtrace.0 file for details
Abort(8) on node 0 (rank 0 in comm 496): application called MPI_Abort(comm=0x84000002, 8) - process 0

Do you know where I can find Backtrace.0? Maybe those details give some clues.

[marchdf]

The Backtrace.0 will be in the run directory of the test. Inside the build directory there's a tests/test_files/name_of_test directory. It will be in there. Do you still have openmp in your build? Or just MPI?

[ArnoldTUB]

Openmp should no longer be in my build (the libraries are no longer installed and AMR_WIND_ENABLE_OPENMP is no longer set so it should be off by default)

[marchdf]

Ok I think I am able to reproduce. And I now remember what's going on. Pretty sure this is it and I fixed it a while ago but never made it explicit: spack/spack#48042. The gist is that OpenFAST initializes some numbers which are actual FPEs. So you need disable that check in openfast. If you are using spack +fpe-trap should do it. If you are building your own openfast, then you need to disable that flag in cmake. Does that make sense?

Give it a shot and lmk if it works. It worked on my end. I can think of trying to make that automatic when using spack.

[ArnoldTUB]

I am glad you were able to reproduce the error.
I was taking a look at the OpenFAST CMAKE options and couldn't find anything that checks for FPE but FPE_TRAP_ENABLED which is off by default and I did not enable it until now. So are you suggesting to compile OpenFAST with this option enabled?

[marchdf]

Yep that's the one. Set it to true. It's a bit confusing. But basically it tells openfast to "be ok with fpe trapping and not throw a fit when we check for FPE" which really means, "disable FPE trapping in openfast because openfast knowingly does some floating point exception arithmetic".