Add Nexus integration #major (#137)

* Add bluepyemodelnexus

* add bluepyemodelnexus part2

* update nexus api

* update local2nexus example

* update setup

* add emodel simulation example with Nexus

* update access_point

* add githash parameter in export_hoc scripts

* docs: Add morph_modifiers info to README

* fix PlotOptimisation task output

* refine morph_modifers explanation in README

* example: add plot scores in exploit_models notebook

* sleep before updating emw

* migrate utcfromtimestamp to fromtimestamp

* add NexusAccessPoint unit tests

* add NexusForgeAccessPoint unit tests

* update copyright

* add debug log for morphology used when using Nexus

* move export_emodels_nexus to export_emodel module

* readme: minor improvement

* add MEModel, update image attachment, refactor download dir

* verify successful download of each datafile for the resource

* update nexus-forge requirement

* allow custom nexus endpoint address

* fix run_emodel readme

* reintroduce EModel_pipeline_nexus class for legacy support

* add memodel example

* remove gitlab CI

* update readme local2nexus

---------

Co-authored-by: Kiliç Ilkan Fabrice <[email protected]>

.gitignore

@@ -16,6 +16,9 @@ software/
 SOURCES.txt
 *.svg
 *.DS_Store
+nexus_temp/
+.tmp/
+figures/
 
 *~
 *.pyc
@@ -136,6 +139,7 @@ celerybeat-schedule
 venv/
 ENV/
 myvenv/
+myvenv_cli/
 
 # Spyder project settings
 .spyderproject

COPYING

@@ -187,7 +187,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2023 Blue Brain Project / EPFL
+   Copyright 2023-2024 Blue Brain Project / EPFL
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

LICENSE.txt

@@ -4,7 +4,7 @@ The examples are under the CC-BY-NC-SA license, as specified by the
 LICENSE.txt file.
 
 
-Copyright 2023 Blue Brain Project / EPFL
+Copyright 2023-2024 Blue Brain Project / EPFL
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

README.rst

@@ -56,6 +56,7 @@ BluePyEModel can be pip installed with the following command:
 If you do not wish to install all dependencies, specific dependencies can be selected by indicating which ones to install between brackets in place of 'all' (If you want multiple dependencies, they have to be separated by commas). The available dependencies are:
 
 * luigi
+* nexus
 * all
 
 To get started with the E-Model building pipeline
@@ -75,7 +76,7 @@ The pipeline is divided in 6 steps:
 * ``plotting``: reads the models and runs the optimisation protocols and/or validation protocols on them. Then, plots the resulting traces along the e-feature scores and parameter distributions.
 * ``exporting``: read the parameter of the best models and export them in files that can be used either in NEURON or for circuit building.
 
-These six steps are to be run in order as for example validation cannot be run if no models have been stored. Steps "validation", "plotting" and "exporting" are optional. Step "extraction" can also be optional in the case where the file containing the protocols and optimisation targets is created by hand or if it is obtained from an older project.
+These six steps are to be run in order as for example validation cannot be run if no models have been stored. Steps ``validation``, ``plotting`` and ``exporting`` are optional. Step ``extraction`` can also be optional in the case where the file containing the protocols and optimisation targets is created by hand or if it is obtained from an older project.
 
 Schematics of BluePyEModel classes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -91,7 +92,7 @@ This work was supported by funding to the Blue Brain Project, a research center
 Copyright
 ~~~~~~~~~
 
-Copyright (c) 2023 Blue Brain Project/EPFL
+Copyright (c) 2023-2024 Blue Brain Project/EPFL
 
 This work is licensed under `Apache 2.0 <https://www.apache.org/licenses/LICENSE-2.0.html>`_
 

bluepyemodel/__init__.py

@@ -1,7 +1,7 @@
 """Main module of BluePyEModel."""
 
 """
-Copyright 2023, EPFL/Blue Brain Project
+Copyright 2023-2024 Blue Brain Project / EPFL
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

bluepyemodel/access_point/__init__.py

@@ -1,7 +1,7 @@
 """E-model access_point module"""
 
 """
-Copyright 2023, EPFL/Blue Brain Project
+Copyright 2023-2024 Blue Brain Project / EPFL
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -70,12 +70,10 @@ def get_access_point(access_point, emodel, **kwargs):
     brain_region = brain_region.replace("__", " ") if brain_region else None
 
     if access_point == "nexus":
-        try:
-            from bluepyemodelnexus.nexus import NexusAccessPoint
-        except ImportError as exc:
-            raise ImportError(
-                "The internal bluepyemodelnexus package is required to use the Nexus access point."
-            ) from exc
+        from bluepyemodel.access_point.nexus import NexusAccessPoint
+
+        if not kwargs.get("project"):
+            raise ValueError("Nexus project name is required for Nexus access point.")
 
         return NexusAccessPoint(
             emodel=emodel,
@@ -88,7 +86,7 @@ def get_access_point(access_point, emodel, **kwargs):
             synapse_class=kwargs.get("synapse_class", None),
             project=kwargs.get("project", None),
             organisation=kwargs.get("organisation", "bbp"),
-            endpoint=kwargs.get("endpoint", "https://bbp.epfl.ch/nexus/v1"),
+            endpoint=kwargs.get("endpoint", "https://staging.nexus.ocp.bbp.epfl.ch/v1"),
             forge_path=kwargs.get("forge_path", None),
             forge_ontology_path=kwargs.get("forge_ontology_path", None),
             access_token=kwargs.get("access_token", None),

bluepyemodel/access_point/access_point.py

@@ -1,7 +1,7 @@
 """DataAccessPoint class."""
 
 """
-Copyright 2023, EPFL/Blue Brain Project
+Copyright 2023-2024 Blue Brain Project / EPFL
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.