| all-to-all metrics | annotator variance |
|---|---|
 |
 |
This codebase is for analyzing and comparing annotator performance for 3D Electron Microscopy (EM) organelle annotations. These annotations are used as ground truth for machine learning algorithms. In particular, we are interested in comparing annotator trials in an all-to-all way across a variety of annotation metrics so we can better determine inter- and intra-annotator variance. This enables identifying those annotators who may need more training. For these calculations, we use CNNectome.
The code contains a variety of functions that are useful for such analysis. The main four of interest are the following:
copy_data: This function copies annotator data into a new directory organized in a way better suited for the remainder of the analysis code.calculate_all_to_all: This function uses Dask to actually do the all-to-all comparison.create_variance_images: This calculates the variance across all manual annotations on a per-voxel basis.get_neuroglancer_view_of_crop: This provides a neuroglancer link to actually view and compare the annotation data.
More details and a concrete example can be found in the metric_evaluations.ipynb python notebook.
To install and use this repository:
- Clone the repository:
git clone https://github.com/janelia-cosem/annotator-metrics. - Move into the directory:
cd annotator-metrics. - Install the conda environment:
conda env create -f environment.yml. - Activate the conda environment:
conda activate annotator-metrics. - Install the package:
pip install ., or if you want to install it for development,pip install -e ..