From d21db9f452b84b34418c079c3c47190eae3b6c03 Mon Sep 17 00:00:00 2001 From: Pat Gunn Date: Mon, 5 Apr 2021 21:08:12 -0400 Subject: [PATCH 01/10] dev: unreleased version string to 1.8.9 --- VERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VERSION b/VERSION index 1790d3598..53ed4ba51 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.8.8 +1.8.9 From b43c92f167808fe28f0e55241cdfd11c21e4ead1 Mon Sep 17 00:00:00 2001 From: Pat Gunn Date: Tue, 20 Apr 2021 10:13:18 -0400 Subject: [PATCH 02/10] Update author/contact email in setup.py --- setup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py index 37e21178b..fdcb0828f 100755 --- a/setup.py +++ b/setup.py @@ -67,7 +67,7 @@ name='caiman', version=version, author='Andrea Giovannucci, Eftychios Pnevmatikakis, Johannes Friedrich, Valentina Staneva, Ben Deverett, Erick Cobos, Jeremie Kalfon', - author_email='epnevmatikakis@flatironinstitute.org', + author_email='pgunn@flatironinstitute.org', url='https://github.com/flatironinstitute/CaImAn', license='GPL-2', description='Advanced algorithms for ROI detection and deconvolution of Calcium Imaging datasets.', @@ -78,7 +78,7 @@ # 3 - Alpha # 4 - Beta # 5 - Production/Stable - 'Development Status :: 4 - Beta', + 'Development Status :: 5 - Production/Stable', # Indicate who your project is intended for 'Intended Audience :: Researchers', From 084252feee7285a912d02a07773502d5d849ea4f Mon Sep 17 00:00:00 2001 From: Pat Gunn Date: Tue, 20 Apr 2021 10:59:09 -0400 Subject: [PATCH 03/10] Readme: rewrite to be streamlined for most common use --- README.md | 105 +++++++++++++++++++++--------------------------------- 1 file changed, 41 insertions(+), 64 deletions(-) diff --git a/README.md b/README.md index 6ce12ad39..cf2466913 100644 --- a/README.md +++ b/README.md @@ -2,19 +2,44 @@ CaImAn ====== +A Python toolbox for large scale **Ca**lcium **Im**aging data **An**alysis and behavioral analysis. +CaImAn implements a set of essential methods required in the analysis pipeline of large scale calcium imaging data. Fast and scalable algorithms are implemented for motion correction, source extraction, spike deconvolution, and component registration across multiple days. It is suitable for both two-photon and one-photon fluorescence microscopy data, and can be run in both batch and online modes. CaImAn also contains some routines for the analysis of behavior from video cameras. A list of features as well as relevant references can be found [here](https://caiman.readthedocs.io/en/latest/CaImAn_features_and_references.html). -[![Join the chat at https://gitter.im/agiovann/SOURCE_EXTRACTION_PYTHON](https://badges.gitter.im/agiovann/SOURCE_EXTRACTION_PYTHON.svg)](https://gitter.im/agiovann/SOURCE_EXTRACTION_PYTHON?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) +## Requirements +Right now, CaImAn works and is supported on the following platforms: +* Linux on Intel CPUs +* MacOS on Intel CPUs +* Windows on Intel CPUs -A Python toolbox for large scale **Ca**lcium **Im**aging data **An**alysis and behavioral analysis. +16G RAM is strongly recommended, and depending on datasets, 32G or more may be helpful. ARM-based versions of Apple hardware are likely to be eventually supported (although current available systems of that sort have little RAM). -CaImAn implements a set of essential methods required in the analysis pipeline of large scale calcium imaging data. Fast and scalable algorithms are implemented for motion correction, source extraction, spike deconvolution, and component registration across multiple days. It is suitable for both two-photon and one-photon fluorescence microscopy data, and can be run in both batch and online modes. CaImAn also contains some routines for the analysis of behavior from video cameras. A list of features as well as relevant references can be found [here](https://caiman.readthedocs.io/en/latest/CaImAn_features_and_references.html). +CaImAn presently targets Python 3.7. Parts of CaImAn are written in C++, but apart possibly during install, this is not visible to the user. There is also an [older implementation](https://github.com/flatironinstitute/CaImAn-MATLAB) of CaImAn in Matlab (unsupported). That version can be used with [MCMC spike inference](https://github.com/epnev/continuous_time_ca_sampler) + +## Install + +The supported ways to install CaImAn use the Anaconda python distribution. If you do not already have it, first install a 3.x version for your platform from [here](https://docs.conda.io/en/latest/miniconda.html). Familiarise yourself with Conda before going further. + +There are a few supported install methods. + +The easiest (and strongly recommended on Windows) is to use a binary conda package, installed as the environment is built. Install this with 'conda create -n caiman -c conda-forge caiman'. This is suitable for most use, if you don't need to change the internals of the caiman package. You do not need to fetch the source code with this approach. + +Another option is to build it yourself; you will need a working compiler (easy on Linux, fairly easy on OSX, fairly involved on Windows). Clone the sources of this repo, and do a 'pip install .' or 'pip install -e .' The former is a standard install, the latter is more suitable for active development on the caiman sources. + +There are other ways to build/use caiman, but they may get less or no support depending on how different they are. -## Web-based Docs -Documentation for CaImAn (including install instructions) can be found online [here](https://caiman.readthedocs.io/en/master/Overview.html). +More detailed docs on installation can be found [here](./docs/source/Installation.rst). -## Companion paper and related references +After installing the software, the caimanmanager.py script (which will be put in your path on Linux and OSX) is used to unpack datafiles and demos into a directory called caiman\_data. + +## Getting Started + +If you used caimanmanager to unpack the demos and data files, you will find in the caiman\_data folder a set of demos and jupyter notebooks. demo\_pipeline.py and demo\_behavior.py (or their notebook equivalents) are good introductions to the code. + +## Papers and data + +### Main paper A paper explaining most of the implementation details and benchmarking can be found [here](https://elifesciences.org/articles/38173). ``` @@ -35,19 +60,14 @@ CaImAn implements a variety of algorithms for analyzing calcium (and voltage) im If you use this code please cite the corresponding papers where original methods appeared as well the companion paper. -## New: Real-time analysis of microendoscopic 1p data (January 2021) +### Real-time analysis of microendoscopic 1p data -Our online algorithms can be used for newly enabled real-time analysis of live-streaming data. An example for real-time analysis of microendoscopic 1p data is shown in the notebook `demos/notebooks/demo_realtime_cnmfE.ipynb`. +Our online algorithms can be used for real-time analysis of live-streaming data. An example for real-time analysis of microendoscopic 1p data is shown in the notebook `demos/notebooks/demo_realtime_cnmfE.ipynb`. For more information about the approach check the [paper](https://doi.org/10.1371/journal.pcbi.1008565). -## New: Online analysis for microendoscopic 1p data (January 2020) - -We developed two approaches for analyzing streaming microendoscopic 1p data with high speed and low memory requirements. -The first approach (OnACID-E) features a direct implementation of the CNMF-E algorithm in an online setup. An example can be seen in the notebook `demos/notebooks/demo_online_cnmfE.ipynb`. The second approach first extracts the background by modeling it with a simple convolutional neural network (Ring-CNN) and proceeds with the analysis using the OnACID algorithm, see `demos/notebooks/demo_Ring_CNN.ipynb`. +### Analysis pipeline for Voltage Imaging data -## New: Analysis pipeline for Voltage Imaging data (December 2019) - -We recently added VolPy, an analysis pipeline for voltage imaging data. The analysis is based on following objects: +VolPy is an analysis pipeline for voltage imaging data. The analysis is based on following objects: * `MotionCorrect`: An object for motion correction which can be used for both rigid and piece-wise rigid motion correction. * `volparams`: An object for setting parameters of voltage imaging. It can be set and changed easily and is passed into the algorithms. @@ -57,42 +77,15 @@ In order to use VolPy, you must install Keras into your conda environment. You c To see examples of how these methods are used, please consult the `demo_pipeline_voltage_imaging.py` script in the `demos/general` folder. For more information about the approach check the [preprint](https://www.biorxiv.org/content/10.1101/2020.01.02.892323v1). -## New: Installation through conda-forge (August 2019) - -Beginning in August 2019 we have an experimental binary release of the software in the conda-forge package repos. This is intended for people who can use CaImAn as a library, interacting with it as the demos do. It also does not need a compiler. It is not suitable for people intending to change the CaImAn codebase. Comfort with conda is still required. If you wish to use the binary package, you do not need the sources (including this repo) at all. Installation and updating instructions can be found [here](./docs/source/Installation.rst). - -You will still need to use caimanmanager.py afterwards to create a data directory. If you install this way, do not follow any of the other install instructions below. - ## Documentation & Wiki Documentation of the code can be found [here](https://caiman.readthedocs.io/en/master/). -### Installation for behavioral analysis -* Installation on Linux (Windows and MacOS are problematic with anaconda at the moment) - * create a new environment (suggested for safety) and follow the instructions for the calcium imaging installation - * Install spams, as explained [here](http://spams-devel.gforge.inria.fr/). Installation is not straightforward and it might take some trials to get it right. - -## Demos - -* Notebooks: The notebooks provide a simple and friendly way to get into CaImAn and understand its main characteristics. -They are located in the `demos/notebooks`. To launch one of the jupyter notebooks: - - ```bash - source activate CaImAn - jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10 - ``` - and select the notebook from within Jupyter's browser. The argument `--NotebookApp.iopub_data_rate_limit=1.0e10` will prevent any memory issues while plotting on a notebook. - -* demo files are also found in the demos/general subfolder. We suggest trying demo_pipeline.py first as it contains most of the tasks required by calcium imaging. For behavior use demo_behavior.py - -* If you modify the demos to use them for your own data it is recommended that you save them in a different file to avoid file conflicts during updating. - -* If you want to directly launch the python files, your python console still must be in the CaImAn directory. - -## Testing +Other docs: +* [Running CaImAn on a Cluster](docs/CLUSTER.md) +* [Install quirks on some Linux Distributions](docs/README-Distros.md) +* [How CaImAn can use your GPUs](docs/README-GPU.md) -* All diffs must be tested before asking for a pull request. Call ```python caimanmanager.py test``` from outside of your CaImAn folder to look for errors (you need to pass the path to the caimanmanager.py file). - # Main developers: * Eftychios A. Pnevmatikakis, **Flatiron Institute, Simons Foundation** @@ -102,23 +95,7 @@ They are located in the `demos/notebooks`. To launch one of the jupyter notebook A complete list of contributors can be found [here](https://github.com/flatironinstitute/CaImAn/graphs/contributors). - -## Other docs in this repo -* [Running CaImAn on a Cluster](docs/CLUSTER.md) -* [Install quirks on some Linux Distributions](docs/README-Distros.md) -* [How CaImAn can use your GPUs](docs/README-GPU.md) - -## Related packages - -The implementation of this package is developed in parallel with a MATLAB toobox, which can be found [here](https://github.com/epnev/ca_source_extraction). - -Some tools that are currently available in Matlab but have not been ported to CaImAn are - -- [MCMC spike inference](https://github.com/epnev/continuous_time_ca_sampler) - -## Dependencies - -A list of dependencies can be found in the [environment file](https://github.com/flatironinstitute/CaImAn/blob/master/environment.yml). +Currently Pat Gunn and Johannes Friedrich are the most active maintainers. ## Questions, comments, issues @@ -127,7 +104,7 @@ Please use the [gitter chat room](https://gitter.im/agiovann/Constrained_NMF) fo ## Acknowledgements -Special thanks to the following people for letting us use their datasets for our various demo files: +Special thanks to the following people for letting us use their datasets in demo files: * Weijian Yang, Darcy Peterka, Rafael Yuste, Columbia University * Sue Ann Koay, David Tank, Princeton University From 1c410fdacd9e61fccf3b0a611249297e0894479e Mon Sep 17 00:00:00 2001 From: Pat Gunn Date: Tue, 20 Apr 2021 11:04:21 -0400 Subject: [PATCH 04/10] Readme: contact info update --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index cf2466913..bbad6c4b8 100644 --- a/README.md +++ b/README.md @@ -100,7 +100,11 @@ Currently Pat Gunn and Johannes Friedrich are the most active maintainers. ## Questions, comments, issues -Please use the [gitter chat room](https://gitter.im/agiovann/Constrained_NMF) for questions and comments and create an issue for any bugs you might encounter. +For support, you can create a Github issue describing any bugs you wish to report, or any feature requests you may have. + +You may also use the [gitter chat room](https://gitter.im/agiovann/Constrained_NMF) for discussion. + +Finally, you may reach out via email to one of the primary maintainers (above). ## Acknowledgements From ea8bb6d4b8ecb43cb322567089ef48d6bc9d5e45 Mon Sep 17 00:00:00 2001 From: Pat Gunn Date: Tue, 20 Apr 2021 11:53:43 -0400 Subject: [PATCH 05/10] README: Add Changlia, another Volpy paper --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index bbad6c4b8..b561b6d89 100644 --- a/README.md +++ b/README.md @@ -77,6 +77,8 @@ In order to use VolPy, you must install Keras into your conda environment. You c To see examples of how these methods are used, please consult the `demo_pipeline_voltage_imaging.py` script in the `demos/general` folder. For more information about the approach check the [preprint](https://www.biorxiv.org/content/10.1101/2020.01.02.892323v1). +There is also a [general paper](https://journals.plos.org/ploscompbiol/article/comments?id=10.1371/journal.pcbi.1008806) on this pipeline + ## Documentation & Wiki Documentation of the code can be found [here](https://caiman.readthedocs.io/en/master/). @@ -91,6 +93,7 @@ Other docs: * Eftychios A. Pnevmatikakis, **Flatiron Institute, Simons Foundation** * Andrea Giovannucci, **University of North Carolina, Chapel Hill** * Johannes Friedrich, **Flatiron Institute, Simons Foundation** +* Changlia Cai, **University of North Carolina, Chapel Hill** * Pat Gunn, **Flatiron Institute, Simons Foundation** A complete list of contributors can be found [here](https://github.com/flatironinstitute/CaImAn/graphs/contributors). From 62ae7bc8a74ea0537249c05cf283052eb07ad127 Mon Sep 17 00:00:00 2001 From: Pat Gunn Date: Wed, 21 Apr 2021 16:09:27 -0400 Subject: [PATCH 06/10] EXPERIMENTAL: modify environment.yml to allow for python 3.8, newer tensorflow --- environment.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/environment.yml b/environment.yml index 881742ac0..900d1cf40 100644 --- a/environment.yml +++ b/environment.yml @@ -1,12 +1,12 @@ channels: - conda-forge dependencies: -- python=3.7 +- python >=3.7 - bokeh - coverage - cython - future -- h5py=2.10.0 +- h5py >=2.10.0 - holoviews - ipykernel - ipython From 47775661b641c7bf3775aab240a877456be4a323 Mon Sep 17 00:00:00 2001 From: Pat Gunn Date: Mon, 3 May 2021 12:10:41 -0400 Subject: [PATCH 07/10] update_temporal_components(): Small docs changes, partially address #867 --- caiman/source_extraction/cnmf/temporal.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/caiman/source_extraction/cnmf/temporal.py b/caiman/source_extraction/cnmf/temporal.py index 0183f7f8e..8c55958d6 100644 --- a/caiman/source_extraction/cnmf/temporal.py +++ b/caiman/source_extraction/cnmf/temporal.py @@ -144,6 +144,12 @@ def update_temporal_components(Y, A, b, Cin, fin, bl=None, c1=None, g=None, sn=N C: np.ndarray matrix of temporal components (K x T) + A: np.ndarray + updated A + + b: np.array + updated estimate + f: np.array vector of temporal background (length T) @@ -156,10 +162,10 @@ def update_temporal_components(Y, A, b, Cin, fin, bl=None, c1=None, g=None, sn=N c1: float same as input - g: float + sn: float same as input - sn: float + g: float same as input YrA: np.ndarray From 75038eb75bfdc3de3d69f1ff372fdb7eea59899d Mon Sep 17 00:00:00 2001 From: Pat Gunn Date: Mon, 3 May 2021 12:17:30 -0400 Subject: [PATCH 08/10] update other environment files to unpin things --- environment-minimal.yml | 2 +- requirements.txt | 2 +- requirements_gpu.txt | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/environment-minimal.yml b/environment-minimal.yml index df772594d..a0198e865 100644 --- a/environment-minimal.yml +++ b/environment-minimal.yml @@ -1,7 +1,7 @@ channels: - conda-forge dependencies: -- python=3.7 +- python=3.8 - cython - future - h5py diff --git a/requirements.txt b/requirements.txt index 0c135393a..b5b252845 100644 --- a/requirements.txt +++ b/requirements.txt @@ -22,7 +22,7 @@ pyqtgraph scikit-image scikit-learn scipy -tensorflow==1.15.4 +tensorflow tifffile tk tqdm diff --git a/requirements_gpu.txt b/requirements_gpu.txt index 6b7621630..0193c6458 100644 --- a/requirements_gpu.txt +++ b/requirements_gpu.txt @@ -22,8 +22,8 @@ pyqtgraph scikit-image scikit-learn scipy -tensorflow-gpu==2.2.1 -tensorflow-addons==0.10.0 +tensorflow-gpu>=2.2.1 +tensorflow-addons>=0.10.0 tifffile tk tqdm From 0ee62c0f8dc8d51a0cd25a0c80a9bc6739403920 Mon Sep 17 00:00:00 2001 From: Pat Gunn Date: Mon, 3 May 2021 12:21:35 -0400 Subject: [PATCH 09/10] to fix merge conflict --- requirements_gpu.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/requirements_gpu.txt b/requirements_gpu.txt index 0193c6458..e2138cddb 100644 --- a/requirements_gpu.txt +++ b/requirements_gpu.txt @@ -22,8 +22,8 @@ pyqtgraph scikit-image scikit-learn scipy -tensorflow-gpu>=2.2.1 -tensorflow-addons>=0.10.0 +tensorflow-gpu +tensorflow-addons tifffile tk tqdm From 5c43de03bf793f849393485f8bbd327716daa49e Mon Sep 17 00:00:00 2001 From: Pat Gunn Date: Mon, 3 May 2021 12:22:24 -0400 Subject: [PATCH 10/10] more merge conflicts --- requirements_gpu.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements_gpu.txt b/requirements_gpu.txt index e2138cddb..2b0f1bee8 100644 --- a/requirements_gpu.txt +++ b/requirements_gpu.txt @@ -22,7 +22,7 @@ pyqtgraph scikit-image scikit-learn scipy -tensorflow-gpu +tensorflow tensorflow-addons tifffile tk