This Repository has been used in a project for the Deep Learning for Computer Vision Course from opencampus.sh

The task was to count the amount of bees in a hive. This was done to determine how healthy the hive is and if different sources of food help bees to sustain better.

This has been done in order to use the repository for this project:

• modified the requirements.txt file to use the correct versions of the packages that have to be imported and adjusted the training to work with our image sizes
• modified the labeling tool to be able to move within the image with w,a,s,d to navigate, as the images were much bigger than the screen resolution
• modified training to reload the latest checkpoint and continue training from there

Open TODOs:

• Preprocess the images to only show the honeycomb before training
• Find memory leak in training
• Use image augmentation to get more consistend results

ORIGINAL DESCRIPTION BELOW

BNN v2

unet style image translation from image of hive entrance to bitmap of location of center of bees.

trained in a semi supervised way on a desktop gpu and deployed to run in real time on the hive using either a raspberry pi using a neural compute stick or a je vois embedded smart camera

see this blog post for more info..

here's an example of predicting bee position on some held out data. the majority of examples trained had ~10 bees per image.

the ability to locate each bee means you can summarise with a count. note the spike around 4pm when the bees at this time of year come back to base.

usage

see run_sample_training_pipeline.sh for an executable end to end smoke test walkthrough of these steps (using sample data)

dependencies

code depends in requirements.txt

sudo apt-get install python3-tk python3-pil python3-pil.imagetk
sudo pip3 install -r requirements.txt

gathering data

the rasp_pi sub directory includes one method of collecting images on a raspberry pi.

labelling

start by using the label_ui.py tool to manually label some images and create a sqlite label.db

the following command starts the labelling tool for some already labelled (by me!) sample data provided with in this repro.

./label_ui.py \
--image-dir sample_data/training/ \
--label-db sample_data/labels.db \
--width 768 --height 1024

hints

• left click to label the center of a bee
• right click to remove the closest label
• press up to toggle labels on / off. this can help in tricky cases.
• use left / right to move between images. it's often helpful when labelling to quickly switch back/forth between images to help distinguish background
• use whatever system your OS provides to zoom in; e.g. in ubuntu super+up / down

you can merge entries from a.db into b.db with merge_db.py

./merge_dbs.py --from-db a.db --into-db b.db

training

before training we materialise a label.db (which is a database of x,y coords) into black and white bitmaps using ./materialise_label_db.py

./materialise_label_db.py \
--label-db sample_data/labels.db \
--directory sample_data/labels/ \
--width 768 --height 1024

we can visualise the training data with data.py. this will generate a number of test*png files with the input data on the left (with data augmentation) and the output labels on the right.

./data.py \
--image-dir sample_data/training/ \
--label-dir sample_data/labels/ \
--width 768 --height 1024

train with train.py.

run denotes the subdirectory for ckpts and tensorboard logs; e.g. --run r12 checkpoints under ckpts/r12/ and logs under tb/r12.

use --help to get complete list of options including model config, defining validation data and stopping conditions.

e.g. to train for a short time on sample_data run the following... (for a more realistic result we'd want to train for many more steps on much more data)

./train.py \
--run r12 \
--steps 300 \
--train-steps 50 \
--train-image-dir sample_data/training/ \
--test-image-dir sample_data/test/ \
--label-dir sample_data/labels/ \
--width 768 --height 1024

progress can be visualised with tensorboard (serves at localhost:6006)

tensorboard --log-dir tb

inference

predictions can be run with predict.py. to specifiy what type of output set one of the following...

• --output-label-db to create a label db; this can be merged with a human labelled db, using ./merge_dbs.py for semi supervised learning
• --export-pngs centroids to export output bitmaps equivalent as those made by ./materialise_label_db.py
• --export-pngs predictions to export explicit model output (i.e. before connected components post processing)

NOTE: given the above step that only runs a short period on a small dataset we DON'T expect this to give a great result; these instructions are more included to prove the plumbing works...

./predict.py \
--run r12 \
--image-dir sample_data/unlabelled \
--output-label-db sample_predictions.db \
--export-pngs predictions

output predictions can be compared to labelled data to calculate precision recall. (we deem a detection correct if it is within a thresholded distance from a label)

./compare_label_dbs.py --true-db ground_truth.db --predicted-db predictions.db
precision 0.936  recall 0.797  f1 0.861

running on compute stick

after loads of hacking i did eventually get this running on the compute stick. some info is included in this issue with additional background info in this forum post

[update] given how long ago this was, i'm hoping things are easier these days :D

some available datasets

• Jonathan Byrne's to-bee-or-not-to-bee kaggle dataset