OpenMPI And Slurm Won't Play Nice

[westobin]

My institution's cluster has recently been upgraded to Slurm 24.0.5 and this appears to conflict with OpenMPI (MPI4PY), and therefore broke my installation of PHOEBE 2.4.14.

I get the following error in my log:

[x1000c2s1b1n0:156995] OPAL ERROR: Not initialized in file pmix3x_client.c at line 112
--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.

  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[x1000c2s1b1n0:156995] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
srun: error: nid0007: task 0: Exited with exit code 1
srun: Terminating StepId=3500615.0

This failure only occurs when using the “srun” command within the Slurm Batch Script. The failure occurs upon importing phoebe. Outside of Slurm, everything appears to run as expected.

Setting the environment variable (export PHOEBE_ENABLE_MPI=FALSE) doesn't change the behavior.

While I am working with the system admins at my university, I thought I would see if there were any other ideas about how to overcome this issue.

Thanks for any advice you can offer!

• Wes

[aprsa]

Hi Wes,

Ask your admins to recompile openmpi with slurm support (--with-slurm option to ./configure). See here for details:

https://www.open-mpi.org/faq/?category=slurm

If they can't do it, you should be able to do it locally, but that may be tricky to do across nodes.