diff --git a/.github/CODE_OF_CONDUCT.md b/.github/CODE_OF_CONDUCT.md index 167c20e..8a8d648 100644 --- a/.github/CODE_OF_CONDUCT.md +++ b/.github/CODE_OF_CONDUCT.md @@ -10,19 +10,19 @@ In the interest of fostering an open and welcoming environment, we as contributo Examples of behavior that contributes to creating a positive environment include: -* Using welcoming and inclusive language -* Being respectful of differing viewpoints and experiences -* Gracefully accepting constructive criticism -* Focusing on what is best for the community -* Showing empathy towards other community members +- Using welcoming and inclusive language +- Being respectful of differing viewpoints and experiences +- Gracefully accepting constructive criticism +- Focusing on what is best for the community +- Showing empathy towards other community members Examples of unacceptable behavior by participants include: -* The use of sexualized language or imagery and unwelcome sexual attention or advances -* Trolling, insulting/derogatory comments, and personal or political attacks -* Public or private harassment -* Publishing others' private information, such as a physical or electronic address, without explicit permission -* Other conduct which could reasonably be considered inappropriate in a professional setting +- The use of sexualized language or imagery and unwelcome sexual attention or advances +- Trolling, insulting/derogatory comments, and personal or political attacks +- Public or private harassment +- Publishing others' private information, such as a physical or electronic address, without explicit permission +- Other conduct which could reasonably be considered inappropriate in a professional setting ## Our Responsibilities diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 2aead1d..6cf0b8e 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -7,101 +7,113 @@ Developing Open Source is great fun! =) Here's the long and short of it: 1. Develop your contribution: - * Pull the latest changes from upstream:: - ``` + + - Pull the latest changes from upstream: + + ```bash git checkout master git pull upstream master ``` - * Create a branch for the feature you want to work on. Since the branch name will appear in the merge message, use a sensible name such as 'transform-speedups':: - ``` + + - Create a branch for the feature you want to work on. Since the branch name will appear in the merge message, use a sensible name such as 'transform-speedups': + ```bash git checkout -b transform-speedups ``` - * Commit locally as you progress (``git add`` and ``git commit``) + - Commit locally as you progress (`git add` and `git commit`) + 1. To submit your contribution: - * Push your changes back to your fork on GitHub:: - ``` + - Push your changes back to your fork on GitHub: + ```bash git push origin transform-speedups ``` - * Enter your GitHub username and password (repeat contributors or advanced users can remove this step by connecting to GitHub with SSH. See detailed instructions below if desired). - * Go to GitHub. The new branch will show up with a green Pull Request button - click it. + - Enter your GitHub username and password (repeat contributors or advanced users can remove this step by connecting to GitHub with SSH. See detailed instructions below if desired). + - Go to GitHub. The new branch will show up with a green Pull Request button - click it. + 1. Review process: - * Reviewers (the other developers and interested community members) will write inline and/or general comments on your Pull Request (PR) to help you improve its implementation, documentation and style. Every single developer working on the project has their code reviewed, and we've come to see it as friendly conversation from which we all learn and the overall code quality benefits. Therefore, please don't let the review discourage you from contributing: its only aim is to improve the quality of project, not to criticize (we are, after all, very grateful for the time you're donating!). - * To update your pull request, make your changes on your local repository and commit. As soon as those changes are pushed up (to the same branch as before) the pull request will update automatically. - * `Travis-CI `__, a continuous integration service, is triggered after each Pull Request update to build the code, run unit tests, measure code coverage and check coding style (PEP8) of your branch. The Travis tests must pass before your PR can be merged. If Travis fails, you can find out why by clicking on the "failed" icon (red cross) and inspecting the build and test log. - * A pull request must be approved by two core team members before merging. -## Guidelines + - Reviewers (the other developers and interested community members) will write inline and/or general comments on your Pull Request (PR) to help you improve its implementation, documentation and style. Every single developer working on the project has their code reviewed, and we've come to see it as friendly conversation from which we all learn and the overall code quality benefits. Therefore, please don't let the review discourage you from contributing: its only aim is to improve the quality of project, not to criticize (we are, after all, very grateful for the time you're donating!). + - To update your pull request, make your changes on your local repository and commit. As soon as those changes are pushed up (to the same branch as before) the pull request will update automatically. + - [Travis-CI](http://travis-ci.org/), a continuous integration service, is triggered after each Pull Request update to build the code, run unit tests, measure code coverage and check coding style (PEP8) of your branch. The Travis tests must pass before your PR can be merged. If Travis fails, you can find out why by clicking on the "failed" icon (red cross) and inspecting the build and test log. + - A pull request must be approved by two core team members before merging. -* All code should have tests (see `test coverage`_ below for more details). -* All code should be documented, to the same - `standard `_ as NumPy and SciPy. -* For new functionality, always add an example to the gallery. -* No changes are ever committed without review and approval by two core team members. **Never merge your own pull request.** -* Examples in the gallery should have a maximum figure width of 8 inches. +## Guidelines +- All code should have tests (see `test coverage`\_ below for more details). +- All code should be documented, to the same + [standard](https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt#docstring-standard) as NumPy and SciPy. +- For new functionality, always add an example to the gallery. +- No changes are ever committed without review and approval by two core team members. **Never merge your own pull request.** +- Examples in the gallery should have a maximum figure width of 8 inches. ## Stylistic Guidelines -* Set up your editor to remove trailing whitespace. Follow `PEP08 `__. Check code with pyflakes / flake8. -* Use numpy data types instead of strings (``np.uint8`` instead of ``"uint8"``). -* Use the following import conventions:: - ``` - import numpy as np - import matplotlib.pyplot as plt - from scipy import ndimage as ndi +- Set up your editor to remove trailing whitespace. Follow [PEP08](http://www.python.org/dev/peps/pep-0008/). Check code with pyflakes / flake8. +- Use numpy data types instead of strings (`np.uint8` instead of `"uint8"`). +- Use the following import conventions: + ```python + import numpy as np + import matplotlib.pyplot as plt + from scipy import ndimage as ndi - cimport numpy as cnp # in Cython code + cimport numpy as cnp # in Cython code ``` -* When documenting array parameters, use ``image : (M, N) ndarray`` and then refer to ``M`` and ``N`` in the docstring, if necessary. -* Refer to array dimensions as (plane), row, column, not as x, y, z. See :ref:`Coordinate conventions ` in the user guide for more information. -* Functions should support all input image dtypes. Use utility functions such as ``img_as_float`` to help convert to an appropriate type. The output format can be whatever is most efficient. This allows us to string together several functions into a pipeline -* Use ``Py_ssize_t`` as data type for all indexing, shape and size variables in C/C++ and Cython code. -* Use relative module imports, i.e. ``from .._shared import xyz`` rather than ``from skimage._shared import xyz``. -* Wrap Cython code in a pure Python function, which defines the API. This improves compatibility with code introspection tools, which are often not aware of Cython code. -* For Cython functions, release the GIL whenever possible, using - ``with nogil:``. - +- When documenting array parameters, use `image : (M, N) ndarray` and then refer to `M` and `N` in the docstring, if necessary. +- Refer to array dimensions as (plane), row, column, not as x, y, z. See :ref:`Coordinate conventions ` in the user guide for more information. +- Functions should support all input image dtypes. Use utility functions such as `img_as_float` to help convert to an appropriate type. The output format can be whatever is most efficient. This allows us to string together several functions into a pipeline +- Use `Py_ssize_t` as data type for all indexing, shape and size variables in C/C++ and Cython code. +- Use relative module imports, i.e. `from .._shared import xyz` rather than `from skimage._shared import xyz`. +- Wrap Cython code in a pure Python function, which defines the API. This improves compatibility with code introspection tools, which are often not aware of Cython code. +- For Cython functions, release the GIL whenever possible, using + `with nogil:`. ## Testing This package has an extensive test suite that ensures correct execution on your system. The test suite has to pass before a pull request can be merged, and tests should be added to cover any modifications to the code base. -We make use of the `pytest `__ testing framework, with tests located in the various ``tests`` folders. +We make use of the [pytest](https://docs.pytest.org/en/latest/) testing framework, with tests located in the various `tests` folders. -To use ``pytest``, ensure that Cython extensions are built and that -the library is installed in development mode:: -``` - $ pip install -e . -``` -Now, run all tests using:: -``` - $ PYTHONPATH=. pytest +To use `pytest`, ensure that Cython extensions are built and that +the library is installed in development mode: + +```bash +$ pip install -e . ``` -Use ``--doctest-modules`` to run doctests. -For example, run all tests and all doctests using:: + +Now, run all tests using: + +```bash +$ PYTHONPATH=. pytest ``` - $ PYTHONPATH=. pytest --doctest-modules --with-xunit --with-coverage + +Use `--doctest-modules` to run doctests. +For example, run all tests and all doctests using: + +```bash +$ PYTHONPATH=. pytest --doctest-modules --with-xunit --with-coverage ``` ## Test coverage Tests for a module should ideally cover all code in that module, i.e., statement coverage should be at 100%. -To measure the test coverage, install `pytest-cov `__ (using ``easy_install pytest-cov``) and then run:: -``` - $ coverage report +To measure the test coverage, install [pytest-cov](http://pytest-cov.readthedocs.io/en/latest/) (using `easy_install pytest-cov`) and then run: + +```bash +$ coverage report ``` + This will print a report with one line for each file in `skimage`, -detailing the test coverage:: +detailing the test coverage: + ``` - Name Stmts Exec Cover Missing - -------------------------------------------------------------- - package/module1 77 77 100% - package/__init__ 1 1 100% - ... +Name Stmts Exec Cover Missing +-------------------------------------------------------------- +package/module1 77 77 100% +package/__init__ 1 1 100% +... ``` ## Bugs -Please `report bugs on GitHub `_. +Please [report bugs on GitHub](https://github.com/Borda/pyImRegBenchmark/issues). diff --git a/.github/ISSUE_TEMPLATE/Bug_report.md b/.github/ISSUE_TEMPLATE/Bug_report.md index 4d516f1..8479e25 100644 --- a/.github/ISSUE_TEMPLATE/Bug_report.md +++ b/.github/ISSUE_TEMPLATE/Bug_report.md @@ -4,12 +4,12 @@ about: Create a report to help us improve title: '' labels: bug assignees: '' - --- ## 🐛 Bug ### Description + - ### Way to reproduce If reporting a bug, please include the following important information: + - [ ] Code example - [ ] Relevant images (if any) - Operating system and version: ... diff --git a/.github/ISSUE_TEMPLATE/Feature_request.md b/.github/ISSUE_TEMPLATE/Feature_request.md index 3350917..ed24db8 100644 --- a/.github/ISSUE_TEMPLATE/Feature_request.md +++ b/.github/ISSUE_TEMPLATE/Feature_request.md @@ -4,10 +4,10 @@ about: Suggest an idea for this project title: '' labels: enhancement, help wanted assignees: '' - --- ## 🚀 Feature + ### Motivation diff --git a/.github/ISSUE_TEMPLATE/Question.md b/.github/ISSUE_TEMPLATE/Question.md index 47844e9..560d937 100644 --- a/.github/ISSUE_TEMPLATE/Question.md +++ b/.github/ISSUE_TEMPLATE/Question.md @@ -4,12 +4,12 @@ about: Asking a questions about... title: '' labels: question assignees: '' - --- ## ❓ Questions and Help ### Before asking: + ---- +______________________________________________________________________ _This project/framework is the key component of **[Automatic Non-rigid Histological Image Registration (ANHIR)](https://anhir.grand-challenge.org)** challenge hosted at [ISBI 2019](https://biomedicalimaging.org/2019/challenges/) conference. The related discussion is hosted on [forum.image.sc](https://forum.image.sc/tags/anhir)._ ---- +______________________________________________________________________ The project contains a set of sample images with related landmark annotations and experimental evaluation of state-of-the-art image registration methods. @@ -44,57 +44,58 @@ For more information about annotation creation and landmarks handling, we refer The dataset is defined by a CSV file containing paths to target and sensed image and their related landmarks _(see `./data-images/pairs-imgs-lnds_mix.csv`)_. With the change of the cover table, the benchmarks can be used for any other image dataset. - ## Features List of the main/key features of this package: -* **automatic** execution of image registration of a sequence of image pairs -* integrated **evaluation** of registration performances using Target Registration Error (TRE) -* integrated **visualization** of performed registration -* running several image registration experiment in **parallel** -* **resuming** unfinished sequence of registration benchmark -* handling around dataset and **creating own experiments** -* using basic **image pre-processing** - normalizing -* rerun evaluation and visualisation for finished experiments - +- **automatic** execution of image registration of a sequence of image pairs +- integrated **evaluation** of registration performances using Target Registration Error (TRE) +- integrated **visualization** of performed registration +- running several image registration experiment in **parallel** +- **resuming** unfinished sequence of registration benchmark +- handling around dataset and **creating own experiments** +- using basic **image pre-processing** - normalizing +- rerun evaluation and visualisation for finished experiments ## Structure The project contains the following folders: -* `benchmarks` - package with benchmark & template and general useful utils - * `utilities` - useful tools and functions -* `bm_ANHIR` - scripts related directly to ANHIR challenge -* `bm_dataset` - package handling dataset creation and servicing -* `bm_experiments` - package with particular benchmark experiments -* `configs` - configs for registration methods -* `data-images` - folder with input sample data - * `images` - sample image pairs (reference and sensed one) - * `landmarks` - related landmarks to images in previous folder - * `lesions_` - samples of histology tissue with annotation - * `rat-kidney_` - samples of histology tissue with annotation -* `docs` - sphinx documentation and references -* `scripts` - scripts and macros (ImageJ/Python/Rscript) for registration and work around -* `tests` - package testing and test requirements - +- `benchmarks` - package with benchmark & template and general useful utils + - `utilities` - useful tools and functions +- `bm_ANHIR` - scripts related directly to ANHIR challenge +- `bm_dataset` - package handling dataset creation and servicing +- `bm_experiments` - package with particular benchmark experiments +- `configs` - configs for registration methods +- `data-images` - folder with input sample data + - `images` - sample image pairs (reference and sensed one) + - `landmarks` - related landmarks to images in previous folder + - `lesions_` - samples of histology tissue with annotation + - `rat-kidney_` - samples of histology tissue with annotation +- `docs` - sphinx documentation and references +- `scripts` - scripts and macros (ImageJ/Python/Rscript) for registration and work around +- `tests` - package testing and test requirements ## Installation This package can be simply install using pip running + ```bash pip install https://github.com/Borda/BIRL/archive/master.zip ``` + or via `setuptools` running from a local folder + ```bash python setup.py install ``` + For installing some particular version/releases use following links _https://github.com/Borda/BIRL/archive/v0.2.3.zip_ where the numbers match desired version (see package [releases](https://github.com/Borda/BIRL/releases)). ---- +______________________________________________________________________ ## Before benchmarks (pre-processing) @@ -133,6 +134,7 @@ python bm_dataset/generate_regist_pairs.py \ ### Customize the images and landmarks We offer a script for scaling images in to particular scales for example + ```bash python bm_dataset/rescale_tissue_images.py \ -i "./data-images/rat-kidney_/scale-5pc/*.jpg" \ @@ -148,11 +150,12 @@ python bm_dataset/rescale_tissue_landmarks.py \ ``` Moreover we developed two additional script for converting large images, handling multiple tissue samples in single image and crop to wide background. - * `bm_dataset/convert_tiff2png.py` converts TIFF or SVS image to PNG in a particular level - * `bm_dataset/split_images_two_tissues.py` splits two tissue samples with clear wide bound in vertical or horizontal direction - * `bm_dataset/crop_tissue_images.py` crops the tissue sample removing wide homogeneous background ---- +- `bm_dataset/convert_tiff2png.py` converts TIFF or SVS image to PNG in a particular level +- `bm_dataset/split_images_two_tissues.py` splits two tissue samples with clear wide bound in vertical or horizontal direction +- `bm_dataset/crop_tissue_images.py` crops the tissue sample removing wide homogeneous background + +______________________________________________________________________ ## Experiments with included methods @@ -165,6 +168,7 @@ Even though this framework is completely customizable we include several image r For each registration method, different experiments can be performed independently using different values of the parameters or image pairs sets. Sample execution of the "empty" benchmark template: + ```bash python birl/bm_template.py \ -t ./data-images/pairs-imgs-lnds_mix.csv \ @@ -173,7 +177,9 @@ python birl/bm_template.py \ --preprocessing matching-rgb gray \ --unique --visual ``` + or with relative paths: + ```bash python birl/bm_template.py \ -t ./data-images/pairs-imgs-lnds_histol.csv \ @@ -186,22 +192,26 @@ python birl/bm_template.py \ The general Image Registration benchmarks contain couple required and optional parameters which are shared among 'all' methods/benchmarks. The brief description is following... **Required** parameters: -* `-t`/`--path_table` path to the cover table describing image/landmarks registration pairs -* `-d`/`--path_dataset` path to the dataset folder with images and landmarks -* `-o`/`--path_out` output path for saving results + +- `-t`/`--path_table` path to the cover table describing image/landmarks registration pairs +- `-d`/`--path_dataset` path to the dataset folder with images and landmarks +- `-o`/`--path_out` output path for saving results **Optional** parameters: -* `--preprocessing` offer some image pre-processing before image registration starts, the order defines order of performed operations; the options are `matching-` (where `` is for example `rgb` or `hsv`) equalise source to target image and `gray` converting both images to gray-scale -* `--unique` each experiment has creation stamp included in its name (prevent overwriting experiments with the same method) -* `--visual` generate a simple visualisation of particular image registrations + +- `--preprocessing` offer some image pre-processing before image registration starts, the order defines order of performed operations; the options are `matching-` (where `` is for example `rgb` or `hsv`) equalise source to target image and `gray` converting both images to gray-scale +- `--unique` each experiment has creation stamp included in its name (prevent overwriting experiments with the same method) +- `--visual` generate a simple visualisation of particular image registrations ![preprocessing-hist-matching](assets/Rat-Kidney_histogram-matching.jpg) Measure your computer performance using average execution time on several simple image registrations. The registration consists of loading images, denoising, feature detection, transform estimation and image warping. + ```bash python bm_experiments/bm_comp_perform.py -o ./results ``` + This script generate simple report exported in JSON file on given output path. ### Prepared experimental docker image @@ -234,104 +244,104 @@ docker run --rm -it birl:py3.6 bash For each benchmark experiment, the explanation about how to install and use a particular registration method is given in the documentation. Brief text at the top of each file. -* **[bUnwarpJ](http://imagej.net/BUnwarpJ)** is the [ImageJ](https://imagej.nih.gov/ij/) plugin for elastic registration (optional usage of histogram matching and integration with [Feature Extraction](http://imagej.net/Feature_Extraction)). - ```bash - python bm_experiments/bm_bUnwarpJ.py \ - -t ./data-images/pairs-imgs-lnds_histol.csv \ - -d ./data-images \ - -o ./results \ - -Fiji ~/Applications/Fiji.app/ImageJ-linux64 \ - -cfg ./configs/ImageJ_bUnwarpJ_histol.yaml \ - --preprocessing matching-rgb \ - --visual --unique - ``` -* **[Register Virtual Stack Slices (RVSS)](https://imagej.net/Register_Virtual_Stack_Slices)** is the [ImageJ](https://imagej.nih.gov/ij/) plugin for affine/elastic registration of a sequence of images. - ```bash - python bm_experiments/bm_RVSS.py \ - -t ./data-images/pairs-imgs-lnds_histol.csv \ - -d ./data-images \ - -o ./results \ - -Fiji ~/Applications/Fiji.app/ImageJ-linux64 \ - -cfg ./configs/ImageJ_RVSS_histol.yaml \ - --visual --unique - ``` -* **[elastix](http://elastix.isi.uu.nl/)** is image registration toolkit based on [ITK](http://www.itk.org/) and it consists of a collection of algorithms that are commonly used to solve (medical) image registration problems. For more details see [documentation](http://elastix.isi.uu.nl/download/elastix_manual_v4.8.pdf). - ```bash - python bm_experiments/bm_elastix.py \ - -t ./data-images/pairs-imgs-lnds_histol.csv \ - -d ./data-images \ - -o ./results \ - -elastix ~/Applications/elastix/bin \ - -cfg ./configs/elastix_affine.txt \ - --visual --unique - ``` -* **[rNiftyReg](https://github.com/jonclayden/RNiftyReg)** is an R-native interface to the [NiftyReg image registration library](http://sourceforge.net/projects/niftyreg/) which contains programs to perform rigid, affine and non-linear registration of Nifty or analyse images. _NiftyReg supports max image size 2048._ - ```bash - python bm_experiments/bm_rNiftyReg.py \ - -t ./data-images/pairs-imgs-lnds_histol.csv \ - -d ./data-images \ - -o ./results \ - -R Rscript \ - -script ./scripts/Rscript/RNiftyReg_linear.r \ - --visual --unique - ``` -* **[Advanced Normalization Tools](http://stnava.github.io/ANTs/) (ANTs)** is a medical imaging framework containing state-of-the-art medical image registration and segmentation methods. - ```bash - python bm_experiments/bm_ANTs.py \ - -t ./data-images/pairs-imgs-lnds_anhir.csv \ - -d ./data-images \ - -o ./results \ - -ANTs ~/Applications/antsbin/bin \ - -cfg ./configs/ANTs_SyN.txt - ``` - For illustration see ANTsPy [registration tutorial](https://github.com/ANTsX/ANTsPy/blob/master/tutorials/10minTutorial.ipynb). - ```bash - python bm_experiments/bm_ANTsPy.py \ - -t ./data-images/pairs-imgs-lnds_histol.csv \ - -d ./data-images \ - -o ./results \ - -py python3 \ - -script ./scripts/Python/run_ANTsPy.py \ - --visual --unique - ``` -* **[DROP](https://github.com/biomedia-mira/drop2)** is image registration and motion estimation based on Markov Random Fields. - ```bash - python bm_experiments/bm_DROP2.py \ - -t ./data-images/pairs-imgs-lnds_histol.csv \ - -d ./data-images \ - -o ./results \ - -DROP ~/Applications/DROP2/dropreg \ - -cfg ./configs/DROP2.txt \ - --visual --unique - ``` -* ... +- **[bUnwarpJ](http://imagej.net/BUnwarpJ)** is the [ImageJ](https://imagej.nih.gov/ij/) plugin for elastic registration (optional usage of histogram matching and integration with [Feature Extraction](http://imagej.net/Feature_Extraction)). + ```bash + python bm_experiments/bm_bUnwarpJ.py \ + -t ./data-images/pairs-imgs-lnds_histol.csv \ + -d ./data-images \ + -o ./results \ + -Fiji ~/Applications/Fiji.app/ImageJ-linux64 \ + -cfg ./configs/ImageJ_bUnwarpJ_histol.yaml \ + --preprocessing matching-rgb \ + --visual --unique + ``` +- **[Register Virtual Stack Slices (RVSS)](https://imagej.net/Register_Virtual_Stack_Slices)** is the [ImageJ](https://imagej.nih.gov/ij/) plugin for affine/elastic registration of a sequence of images. + ```bash + python bm_experiments/bm_RVSS.py \ + -t ./data-images/pairs-imgs-lnds_histol.csv \ + -d ./data-images \ + -o ./results \ + -Fiji ~/Applications/Fiji.app/ImageJ-linux64 \ + -cfg ./configs/ImageJ_RVSS_histol.yaml \ + --visual --unique + ``` +- **[elastix](http://elastix.isi.uu.nl/)** is image registration toolkit based on [ITK](http://www.itk.org/) and it consists of a collection of algorithms that are commonly used to solve (medical) image registration problems. For more details see [documentation](http://elastix.isi.uu.nl/download/elastix_manual_v4.8.pdf). + ```bash + python bm_experiments/bm_elastix.py \ + -t ./data-images/pairs-imgs-lnds_histol.csv \ + -d ./data-images \ + -o ./results \ + -elastix ~/Applications/elastix/bin \ + -cfg ./configs/elastix_affine.txt \ + --visual --unique + ``` +- **[rNiftyReg](https://github.com/jonclayden/RNiftyReg)** is an R-native interface to the [NiftyReg image registration library](http://sourceforge.net/projects/niftyreg/) which contains programs to perform rigid, affine and non-linear registration of Nifty or analyse images. _NiftyReg supports max image size 2048._ + ```bash + python bm_experiments/bm_rNiftyReg.py \ + -t ./data-images/pairs-imgs-lnds_histol.csv \ + -d ./data-images \ + -o ./results \ + -R Rscript \ + -script ./scripts/Rscript/RNiftyReg_linear.r \ + --visual --unique + ``` +- **[Advanced Normalization Tools](http://stnava.github.io/ANTs/) (ANTs)** is a medical imaging framework containing state-of-the-art medical image registration and segmentation methods. + ```bash + python bm_experiments/bm_ANTs.py \ + -t ./data-images/pairs-imgs-lnds_anhir.csv \ + -d ./data-images \ + -o ./results \ + -ANTs ~/Applications/antsbin/bin \ + -cfg ./configs/ANTs_SyN.txt + ``` + For illustration see ANTsPy [registration tutorial](https://github.com/ANTsX/ANTsPy/blob/master/tutorials/10minTutorial.ipynb). + ```bash + python bm_experiments/bm_ANTsPy.py \ + -t ./data-images/pairs-imgs-lnds_histol.csv \ + -d ./data-images \ + -o ./results \ + -py python3 \ + -script ./scripts/Python/run_ANTsPy.py \ + --visual --unique + ``` +- **[DROP](https://github.com/biomedia-mira/drop2)** is image registration and motion estimation based on Markov Random Fields. + ```bash + python bm_experiments/bm_DROP2.py \ + -t ./data-images/pairs-imgs-lnds_histol.csv \ + -d ./data-images \ + -o ./results \ + -DROP ~/Applications/DROP2/dropreg \ + -cfg ./configs/DROP2.txt \ + --visual --unique + ``` +- ... Some more image registration methods integrated in ImageJ are listed in [Registration](https://imagej.net/Registration). - ### Add custom registration method The only limitation of adding costume image registration methods that it has to be launched from python script or command line. The new registration benchmark should be inherited from [ImRegBenchmark](birl/benchmark.py) - as for example [BmTemplate](birl/bm_template.py). +as for example [BmTemplate](birl/bm_template.py). The benchmark workflow is the following: + 1. `self._prepare()` prepare the experiment, e.g. create experiment folder, copy configurations, etc. -2. `self._load_data()` the load required data - the experiment cover file -3. `self._run()` perform the sequence of experiments (optionally in parallel) and save experimental results (registration outputs and partial statistic) to common table and optionally do particular visualisation of performed experiments -4. `self._evaluate()` evaluate results of all performed experiments -4. `self._summarise()` summarize and export results from complete benchmark. +1. `self._load_data()` the load required data - the experiment cover file +1. `self._run()` perform the sequence of experiments (optionally in parallel) and save experimental results (registration outputs and partial statistic) to common table and optionally do particular visualisation of performed experiments +1. `self._evaluate()` evaluate results of all performed experiments +1. `self._summarise()` summarize and export results from complete benchmark. General methods that should be overwritten: - * `_check_required_params(...)` verify that all required input parameters are given _[on the beginning of benchmark]_ - * `_prepare_img_registration(...)` if some extra preparation before running own image registrations are needed _[before each image registration experiment]_ - * `_execute_img_registration` execute/perform the image registration, time of this methods is measured as execution time. In case you call external method from command line, just rewrite `_generate_regist_command(...)` which prepare the registration command to be executed, also you can add generating complete registration script/macro if needed; _[core of each image registration experiment]_ - * `_extract_warped_image_landmarks(...)` extract the required warped landmarks or perform landmark warping in this stage if it was not already part of the image registration _[after each image registration experiment]_ - * `_extract_execution_time(...)` optionally extract the execution time from the external script _[after each image registration experiment]_ - * `_clear_after_registration(...)` removing some temporary files generated during image registration _[after each image registration experiment]_ -The new image registration methods should be added to `bm_experiments` folder. +- `_check_required_params(...)` verify that all required input parameters are given _\[on the beginning of benchmark\]_ +- `_prepare_img_registration(...)` if some extra preparation before running own image registrations are needed _\[before each image registration experiment\]_ +- `_execute_img_registration` execute/perform the image registration, time of this methods is measured as execution time. In case you call external method from command line, just rewrite `_generate_regist_command(...)` which prepare the registration command to be executed, also you can add generating complete registration script/macro if needed; _\[core of each image registration experiment\]_ +- `_extract_warped_image_landmarks(...)` extract the required warped landmarks or perform landmark warping in this stage if it was not already part of the image registration _\[after each image registration experiment\]_ +- `_extract_execution_time(...)` optionally extract the execution time from the external script _\[after each image registration experiment\]_ +- `_clear_after_registration(...)` removing some temporary files generated during image registration _\[after each image registration experiment\]_ +The new image registration methods should be added to `bm_experiments` folder. ### Re-evaluate experiment @@ -344,25 +354,26 @@ python bm_experiments/evaluate_experiment.py \ --visual ``` ---- +______________________________________________________________________ ## License The project is using the standard [BSD license](http://opensource.org/licenses/BSD-3-Clause). - ## References For complete references see [bibtex](docs/references.bib). + 1. Borovec, J., Munoz-Barrutia, A., & Kybic, J. (2018). **[Benchmarking of image registration methods for differently stained histological slides](https://www.researchgate.net/publication/325019076_Benchmarking_of_image_registration_methods_for_differently_stained_histological_slides)**. In IEEE International Conference on Image Processing (ICIP) (pp. 3368–3372), Athens. [DOI: 10.1109/ICIP.2018.8451040](https://doi.org/10.1109/ICIP.2018.8451040) -2. Borovec, J. (2019). **[BIRL: Benchmark on Image Registration methods with Landmark validation](https://www.researchgate.net/publication/338291737_BIRL_Benchmark_on_Image_Registration_methods_with_Landmark_validation)**. arXiv preprint [arXiv:1912.13452](https://arxiv.org/abs/1912.13452). -3. Borovec, J., Kybic, J., et al. (2020). **[ANHIR: Automatic Non-rigid Histological Image Registration Challenge](https://ieeexplore.ieee.org/document/9058666)**. In IEEE Transaction on Medical Imaging (TMI). [DOI: 10.1109/TMI.2020.2986331](https://www.doi.org/10.1109/TMI.2020.2986331). +1. Borovec, J. (2019). **[BIRL: Benchmark on Image Registration methods with Landmark validation](https://www.researchgate.net/publication/338291737_BIRL_Benchmark_on_Image_Registration_methods_with_Landmark_validation)**. arXiv preprint [arXiv:1912.13452](https://arxiv.org/abs/1912.13452). +1. Borovec, J., Kybic, J., et al. (2020). **[ANHIR: Automatic Non-rigid Histological Image Registration Challenge](https://ieeexplore.ieee.org/document/9058666)**. In IEEE Transaction on Medical Imaging (TMI). [DOI: 10.1109/TMI.2020.2986331](https://www.doi.org/10.1109/TMI.2020.2986331). ## Appendix - Useful information **Configure local environment** Create your own local environment, for more information see the [User Guide](https://pip.pypa.io/en/latest/user_guide.html), and install dependencies requirements.txt contains a list of packages and can be installed as + ```bash @duda:~$ cd BIRL @duda:~/BIRL$ virtualenv env @@ -370,7 +381,9 @@ Create your own local environment, for more information see the [User Guide](htt (env)@duda:~/BIRL$ pip install -r requirements.txt (env)@duda:~/BIRL$ python ... ``` + and in the end, terminating... + ```bash (env)@duda:~$ deactivate ``` @@ -378,6 +391,7 @@ and in the end, terminating... **Running docString tests** - documentation and samples of doc string on [pymotw](https://pymotw.com/2/doctest/) and [python/docs](https://docs.python.org/2/library/doctest.html) **Listing dataset in command line** + ```bash find . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/" >> dataset.txt ``` diff --git a/bm_ANHIR/automatic-evaluation.md b/bm_ANHIR/automatic-evaluation.md index 7a437d8..1a9b2aa 100644 --- a/bm_ANHIR/automatic-evaluation.md +++ b/bm_ANHIR/automatic-evaluation.md @@ -21,37 +21,41 @@ docker build -t anhir -f Dockerfile . ### Run and Test Run one of following sample registration experiments: - * simulate the ideal registration, assuming having all landmarks - ```bash - python benchmark/bm_template.py \ - -t ~/Medical-data/dataset_ANHIR/images/dataset_medium.csv \ - -d ~/Medical-temp/dataset_ANHIR/images \ - -o ~/Medical-temp/experiments_anhir/ \ - -cfg sample_config.yaml - python bm_experiments/bm_comp_perform.py -o ~/Medical-temp/experiments_anhir/BmTemplate - # remove all registered images - rm ~/Medical-temp/experiments_anhir/BmTemplate/*/*.jpg \ - ~/Medical-temp/experiments_anhir/BmTemplate/*/*.png - ``` - * run bUnwarpJ in ImageJ registration on the real data - ```bash - python bm_experiments/bm_bUnwarpJ.py \ - -t ~/Medical-data/dataset_ANHIR/images/dataset_medium.csv \ - -d ~/Medical-temp/dataset_ANHIR/images \ - -o ~/Medical-temp/experiments_anhir/ \ - --run_comp_benchmark \ - -Fiji ~/Applications/Fiji.app/ImageJ-linux64 \ - -cfg ./configs/ImageJ_bUnwarpJ_histol-1k.txt - # remove all registered images - rm ~/Medical-temp/experiments_anhir/BmUnwarpJ/*/*.jpg \ - ~/Medical-temp/experiments_anhir/BmUnwarpJ/*/*.png - ``` + +- simulate the ideal registration, assuming having all landmarks + ```bash + python benchmark/bm_template.py \ + -t ~/Medical-data/dataset_ANHIR/images/dataset_medium.csv \ + -d ~/Medical-temp/dataset_ANHIR/images \ + -o ~/Medical-temp/experiments_anhir/ \ + -cfg sample_config.yaml + python bm_experiments/bm_comp_perform.py -o ~/Medical-temp/experiments_anhir/BmTemplate + # remove all registered images + rm ~/Medical-temp/experiments_anhir/BmTemplate/*/*.jpg \ + ~/Medical-temp/experiments_anhir/BmTemplate/*/*.png + ``` +- run bUnwarpJ in ImageJ registration on the real data + ```bash + python bm_experiments/bm_bUnwarpJ.py \ + -t ~/Medical-data/dataset_ANHIR/images/dataset_medium.csv \ + -d ~/Medical-temp/dataset_ANHIR/images \ + -o ~/Medical-temp/experiments_anhir/ \ + --run_comp_benchmark \ + -Fiji ~/Applications/Fiji.app/ImageJ-linux64 \ + -cfg ./configs/ImageJ_bUnwarpJ_histol-1k.txt + # remove all registered images + rm ~/Medical-temp/experiments_anhir/BmUnwarpJ/*/*.jpg \ + ~/Medical-temp/experiments_anhir/BmUnwarpJ/*/*.png + ``` Running the docker image with mapped folders + ```bash mkdir submission output ``` + and upload the sample submission to `submission` and run the image + ```bash docker run --rm -it \ --memory=4g \ @@ -60,10 +64,10 @@ docker run --rm -it \ anhir ``` - ### Export Export the created image to be uploaded to the evaluation system. + ```bash # full size image docker save anhir > anhir.tar @@ -75,23 +79,26 @@ docker save anhir | gzip -c > anhir.tar.gz **Browsing** To see your locally build images use: + ```bash docker image ls ``` **Cleaning** In case you fail with some builds, you may need to clean your local storage. + ```bash docker system prune ``` + or [Docker - How to cleanup (unused) resources](https://gist.github.com/bastman/5b57ddb3c11942094f8d0a97d461b430) + ```bash docker images | grep "none" docker rmi $(docker images | grep "none" | awk '/ / { print $3 }') ``` - ## References -* https://evalutils.readthedocs.io/en/latest/usage.html -* https://grand-challengeorg.readthedocs.io/en/latest/evaluation.html +- https://evalutils.readthedocs.io/en/latest/usage.html +- https://grand-challengeorg.readthedocs.io/en/latest/evaluation.html diff --git a/bm_CIMA/README.md b/bm_CIMA/README.md index 534ee30..433eac0 100644 --- a/bm_CIMA/README.md +++ b/bm_CIMA/README.md @@ -8,7 +8,6 @@ This section is strictly limited to image registration experiment on [CIMA datas - **Script**: the execution script is `run-SOTA-experiments.sh` and it perform all experiments - **Results**: the experimental results are exported and zipped per particular dataset scope, the archives are `results_size-.zip` - ## Usage **Reproduce statistic** @@ -20,13 +19,14 @@ Note that with using attached JSON results you do not need to run cells related **Add own method to statistic** You need to run your benchmark on the particular dataset scope, the image oaring are: + - [10k scope](dataset_CIMA_10k.csv) - [full scope](dataset_CIMA_full.csv) Then you can parse just the new results with [evaluation script](../bm_ANHIR/evaluate_submission.py) or execute the parsing cells at the beginning of [scope notebook](../notebooks/CIMA_SOTA-results_scope.ipynb). - ## References For complete references see [bibtex](../docs/references.bib). + 1. Borovec, J. (2019). **BIRL: Benchmark on Image Registration methods with Landmark validation**. arXiv preprint [arXiv:1912.13452.](https://arxiv.org/abs/1912.13452) diff --git a/docs/source/conf.py b/docs/source/conf.py index 6e8b34b..2e9d8ec 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -26,7 +26,7 @@ PATH_ROOT = os.path.abspath(os.path.join(PATH_HERE, PATH_UP)) sys.path.insert(0, os.path.abspath(PATH_ROOT)) -import birl # noqa: E402 +import birl # -- Project information ----------------------------------------------------- diff --git a/require-py27.txt b/require-py27.txt index a431dc0..0174c8d 100644 --- a/require-py27.txt +++ b/require-py27.txt @@ -1,15 +1,15 @@ numpy>=1.9 scipy>=0.10.0 -pandas>=0.17.1 -six>=1.7.3 +pandas>=0.17.1, <1.0 +six>1.7.3 pillow matplotlib>=2.0.2, <3.0.0 # new version does not support py2 tqdm>=4.7.4, <=4.30 # higher fails ascii for py2 pyyaml -scikit-image +scikit-image>=0.13, <0.18 opencv-python-headless<4.3 openslide-python<=1.1.1 nibabel<3.0 -SimpleITK +SimpleITK<2.0 psutil pathos