Fixes for building portable wheel : nrnivmodl-core should be usable from wheel #634

pramodk · 2021-09-08T22:06:44Z

Do not link to MPI libraries if MPI is not enabled
Use CXX compiler set on target machine e.g. via env variable
Perl and Python executable may not be exist in same path.
Check for existence before using them.

Todos:

Check is Perl and Python should be set via wrapper?
Perl is safe. May be only python is worth setting.

CI_BRANCHES:NEURON_BRANCH=epic/gpu_wheel,

* do not propogate NEURON MPI flag if CoreNEURON MPI option is explicitly specified * nrnivmodl uses CORENRNHOME env variable instead of CNRNHOE variable * binwrapper.py sets CORENRNHOME same as NRNHOME * update coreneuron to branch BlueBrain/CoreNeuron#634

bbpbuildbot · 2021-09-08T23:22:10Z

Logfiles from GitLab pipeline #16159 (:white_check_mark:) have been uploaded here!

Status and direct links:

bbpbuildbot · 2021-10-12T16:22:34Z

Logfiles from GitLab pipeline #19667 (:white_check_mark:) have been uploaded here!

Status and direct links:

pramodk · 2021-10-12T23:14:31Z

Still WIP but on BB5 I am testing this with:

   # clone neuron from : https://github.com/neuronsimulator/nrn/pull/1452 i.e. branch `epic/gpu_wheel`

    module load unstable cmake python hpe-mpi flex bison python-dev llvm gcc
    cmake .. -DCMAKE_INSTALL_PREFIX=`pwd`/install \
          -DCORENRN_ENABLE_GPU=OFF -DNRN_ENABLE_INTERVIEWS=OFF -DNRN_ENABLE_RX3D=OFF \
          -DNRN_ENABLE_CORENEURON=ON -DCORENRN_ENABLE_NMODL=OFF -DCORENRN_CLANG_FORMAT=ON \
          -DNRN_ENABLE_MPI_DYNAMIC=ON \
          -DNRN_MPI_DYNAMIC="/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/externals/2021-01-06/linux-rhel7-x86_64/gcc-9.3.0/intel-mpi-2019.8.254-r
          -DCORENRN_CMAKE_FORMAT=ON -DPYTHON_EXECUTABLE=`which python3.8`
    make -j12
    make install

and it should create

$ tree lib/
lib/
|-- libcoreneuron.so
|-- libcorenrnmpi_mpich.so
|-- libcorenrnmpi_mpt.so
|-- libnrniv.so
|-- libnrnmpi_mpich.so
|-- libnrnmpi_mpt.so
|-- librxdmath.so
|-- libscopmath.a

@alkino : still draft but you can see my ongoing changes in 836267d.

bbpbuildbot · 2021-10-13T00:33:30Z

Logfiles from GitLab pipeline #19704 (:no_entry:) have been uploaded here!

Status and direct links:

bbpbuildbot · 2021-10-13T15:36:37Z

Logfiles from GitLab pipeline #19841 (:no_entry:) have been uploaded here!

Status and direct links:

coreneuron/apps/corenrn_parameters.cpp

bbpbuildbot · 2021-10-13T17:57:34Z

Logfiles from GitLab pipeline #19854 (:no_entry:) have been uploaded here!

Status and direct links:

bbpbuildbot · 2021-10-13T21:28:49Z

Logfiles from GitLab pipeline #19878 (:white_check_mark:) have been uploaded here!

Status and direct links:

* do not propogate NEURON MPI flag if CoreNEURON MPI option is explicitly specified * nrnivmodl uses CORENRNHOME env variable instead of CNRNHOE variable * binwrapper.py sets CORENRNHOME same as NRNHOME * update coreneuron to branch BlueBrain/CoreNeuron#634

Update coreneuron to BlueBrain/CoreNeuron#634

bbpbuildbot · 2021-10-14T00:38:49Z

Logfiles from GitLab pipeline #19887 (:white_check_mark:) have been uploaded here!

Status and direct links:

bbpbuildbot · 2021-10-14T19:38:46Z

Logfiles from GitLab pipeline #20108 (:no_entry:) have been uploaded here!

Status and direct links:

coreneuron/apps/corenrn_parameters.hpp

coreneuron/apps/main1.cpp

coreneuron/CMakeLists.txt

bbpbuildbot · 2021-10-18T15:11:18Z

Logfiles from GitLab pipeline #20492 (:no_entry:) have been uploaded here!

Status and direct links:

* Do not link to MPI libraries if MPI is not enabled * Use CXX compiler set on target machine e.g. via env variable * Perl and Python executable may not be exist in same path. Check for existence before using them. Todo: - [ ] Check is Perl and Python should be set via wrapper? Perl is safe. May be only python is worth.

…N_ENABLE_MPI_DYNAMIC

* this is only supported when build as a submodule of neuron to avoid cmake code duplication * leverage -DNRN_MPI_DYNAMIC="a;b;" option of neuron * MPI distribution specific libcorenrnmpi_<name>.so is built and installed under lib folder * TODO: still need to inrerface with NEURON side On BB5 this is tested with: ``` module load unstable cmake python hpe-mpi flex bison python-dev llvm gcc cmake .. -DCMAKE_INSTALL_PREFIX=`pwd`/install \ -DCORENRN_ENABLE_GPU=OFF -DNRN_ENABLE_INTERVIEWS=OFF -DNRN_ENABLE_RX3D=OFF \ -DNRN_ENABLE_CORENEURON=ON -DCORENRN_ENABLE_NMODL=OFF -DCORENRN_CLANG_FORMAT=ON \ -DNRN_ENABLE_MPI_DYNAMIC=ON \ -DNRN_MPI_DYNAMIC="/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/externals/2021-01-06/linux-rhel7-x86_64/gcc-9.3.0/intel-mpi-2019.8.254-rrc5ip/impi/2019.8.254/intel64/include/;/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/externals/2021-01-06/linux-rhel7-x86_64/gcc-9.3.0/hpe-mpi-2.22.hmpt-r52ypu/include;" \ -DCORENRN_CMAKE_FORMAT=ON -DPYTHON_EXECUTABLE=`which python3.8` make -j12 make install ```

* add new CLI option --corenrn-mpi-lib * load & resolve symbols only once * is_quite now passed as argument to nrnmpi_init * remove usage of [dy]ld_library_path

…s wheel wrapper

* change --corenrn-mpi-lib to --mpi-lib * use variable for name of the library and targets

* do not compile mpi library with -acc -gpu flags because it results into nasty segfault * to achieve this, do not add OpenACC flags to CMAKE_CXX_FLAGS instead use targe_compile_options() * Profile interface needs to be updated because some cpp files like for mpi library may not have cuda headers because they no longer use -acc flag to compile. Check existance of ACC or CUDA active.

Linker should use -acc -cuda instead of just -cuda

bbpbuildbot · 2021-10-20T14:52:54Z

Logfiles from GitLab pipeline #20872 (:white_check_mark:) have been uploaded here!

Status and direct links:

olupton

LGTM, just a couple of minor things.

re: the error with building too many things with -acc, do we have an issue to follow up on that?

CMake/OpenAccHelper.cmake

coreneuron/apps/main1.cpp

bbpbuildbot · 2021-10-20T19:11:33Z

Logfiles from GitLab pipeline #20938 (:white_check_mark:) have been uploaded here!

Status and direct links:

pramodk · 2021-10-20T21:39:06Z

re: the error with building too many things with -acc, do we have an issue to follow up on that?

I created #674 for this.

pramodk · 2021-10-20T21:53:08Z

Retest this please

bbpbuildbot · 2021-10-20T22:30:49Z

Logfiles from GitLab pipeline #20945 (:white_check_mark:) have been uploaded here!

Status and direct links:

pramodk · 2021-10-21T22:59:58Z

Retest this please

…n#634) * Do not link to MPI libraries if MPI is not enabled * Use CXX compiler set on target machine e.g. via env variable * Perl and Python executable may not be exist in same path. - Check for existence before using them. - Respect CORENRN_PERLEXE and CORENRN_PYTHONEXE env vars set by neuron's wheel wrapper * neuron cmake option consistency: CORENRN_ENABLE_DYNAMIC_MPI -> CORENRN_ENABLE_MPI_DYNAMIC * Build multiple MPI libraries for dynamic MPI support - this is only supported when build as a submodule of neuron to avoid cmake code duplication - leverage -DNRN_MPI_DYNAMIC="a;b;" option of neuron - MPI distribution specific libcorenrnmpi_<name>.so is built and installed under lib folder * Add CLI option --mpi-lib which can accept name of coreneuron MPI library to load * Load & resolve symbols only once * Cleanup and Update tests * Fixes for segfault at finalise with dynamic MPI - do not compile mpi library with -acc -gpu flags because it results into nasty segfault - to achieve this, do not add OpenACC flags to CMAKE_CXX_FLAGS instead use targe_compile_options() - profile interface needs to be updated because some cpp files like for mpi library may not have cuda headers because they no longer use -acc flag to compile. Check existance of ACC or CUDA active. CoreNEURON Repo SHA: BlueBrain/CoreNeuron@ec0a4d4

pramodk mentioned this pull request Sep 8, 2021

Support for building GPU enabled wheel with CoreNEURON neuronsimulator/nrn#1452

Merged

6 tasks

alkino approved these changes Sep 10, 2021

View reviewed changes

pramodk mentioned this pull request Oct 8, 2021

Make MPI functions dynamically loadable #641

Merged

pramodk force-pushed the pramodk/wheel-fixes branch from fca92bf to e41cb5c Compare October 12, 2021 14:43

pramodk force-pushed the pramodk/wheel-fixes branch 2 times, most recently from 2ac1405 to 7187ac5 Compare October 13, 2021 16:07

alkino reviewed Oct 13, 2021

View reviewed changes

coreneuron/apps/corenrn_parameters.cpp Outdated Show resolved Hide resolved

pramodk force-pushed the pramodk/wheel-fixes branch from 1e5bd1b to 0f4ab0e Compare October 13, 2021 19:34

pramodk force-pushed the pramodk/wheel-fixes branch from 0f4ab0e to 2ffcc0b Compare October 13, 2021 23:45

pramodk requested a review from olupton October 13, 2021 23:45

pramodk added a commit to neuronsimulator/nrn that referenced this pull request Oct 13, 2021

Coreneuron dynamic MPI integration

fec3a76

Update coreneuron to BlueBrain/CoreNeuron#634

alexsavulescu approved these changes Oct 14, 2021

View reviewed changes

alkino reviewed Oct 18, 2021

View reviewed changes

coreneuron/apps/corenrn_parameters.hpp Outdated Show resolved Hide resolved

alkino reviewed Oct 18, 2021

View reviewed changes

coreneuron/apps/main1.cpp Outdated Show resolved Hide resolved

coreneuron/CMakeLists.txt Outdated Show resolved Hide resolved

pramodk force-pushed the pramodk/wheel-fixes branch from 011d528 to ef48628 Compare October 18, 2021 13:39

pramodk added 8 commits October 20, 2021 16:05

neuron cmake option consistency: CORENRN_ENABLE_DYNAMIC_MPI -> CORENR…

18cdf99

…N_ENABLE_MPI_DYNAMIC

Connect dynamic MPI with neuron

84c14d3

* add new CLI option --corenrn-mpi-lib * load & resolve symbols only once * is_quite now passed as argument to nrnmpi_init * remove usage of [dy]ld_library_path

Respect CORENRN_PERLEXE and CORENRN_PYTHONEXE env vars set by neuron'…

ed629af

…s wheel wrapper

Cleanup and address review comments

8df95db

* change --corenrn-mpi-lib to --mpi-lib * use variable for name of the library and targets

Remove extra variable CORENRN_ACC_FLAGS

60b33ba

Linker should use -acc -cuda instead of just -cuda

Address code review comments

abd397e

pramodk force-pushed the pramodk/wheel-fixes branch from ef48628 to abd397e Compare October 20, 2021 14:06

olupton reviewed Oct 20, 2021

View reviewed changes

CMake/OpenAccHelper.cmake Show resolved Hide resolved

coreneuron/apps/main1.cpp Outdated Show resolved Hide resolved

Update mod2c to latest master

5afe8a2

Merge branch 'master' into pramodk/wheel-fixes

045c3dd

pramodk mentioned this pull request Oct 20, 2021

Improvements for GPU binary wheels #672

Open

5 tasks

Address review comments

5106f40

olupton approved these changes Oct 21, 2021

View reviewed changes

pramodk merged commit ec0a4d4 into master Oct 21, 2021

pramodk deleted the pramodk/wheel-fixes branch October 21, 2021 23:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes for building portable wheel : nrnivmodl-core should be usable from wheel #634

Fixes for building portable wheel : nrnivmodl-core should be usable from wheel #634

pramodk commented Sep 8, 2021 •

edited

Loading

bbpbuildbot commented Sep 8, 2021

bbpbuildbot commented Oct 12, 2021

pramodk commented Oct 12, 2021 •

edited

Loading

bbpbuildbot commented Oct 13, 2021

bbpbuildbot commented Oct 13, 2021

bbpbuildbot commented Oct 13, 2021

bbpbuildbot commented Oct 13, 2021

bbpbuildbot commented Oct 14, 2021

bbpbuildbot commented Oct 14, 2021

bbpbuildbot commented Oct 18, 2021

bbpbuildbot commented Oct 20, 2021

olupton left a comment

bbpbuildbot commented Oct 20, 2021

pramodk commented Oct 20, 2021

pramodk commented Oct 20, 2021

bbpbuildbot commented Oct 20, 2021

pramodk commented Oct 21, 2021

Fixes for building portable wheel : nrnivmodl-core should be usable from wheel #634

Fixes for building portable wheel : nrnivmodl-core should be usable from wheel #634

Conversation

pramodk commented Sep 8, 2021 • edited Loading

Todos:

bbpbuildbot commented Sep 8, 2021

bbpbuildbot commented Oct 12, 2021

pramodk commented Oct 12, 2021 • edited Loading

bbpbuildbot commented Oct 13, 2021

bbpbuildbot commented Oct 13, 2021

bbpbuildbot commented Oct 13, 2021

bbpbuildbot commented Oct 13, 2021

bbpbuildbot commented Oct 14, 2021

bbpbuildbot commented Oct 14, 2021

bbpbuildbot commented Oct 18, 2021

bbpbuildbot commented Oct 20, 2021

olupton left a comment

Choose a reason for hiding this comment

bbpbuildbot commented Oct 20, 2021

pramodk commented Oct 20, 2021

pramodk commented Oct 20, 2021

bbpbuildbot commented Oct 20, 2021

pramodk commented Oct 21, 2021

pramodk commented Sep 8, 2021 •

edited

Loading

pramodk commented Oct 12, 2021 •

edited

Loading