-
Notifications
You must be signed in to change notification settings - Fork 1
For People Looking to Run the Pipeline
Please read this setup guide
Currently, the code is setup on purves, and people in the bee lab should be able to just execute the code directly on purves by logging in via ssh. In order to run the pipeline you need to be able to save and download packages larger then your home directory. Please install miniconda3 using this guide written by Tom Fu. Then install the dependencies listed on the main README.
The pipeline code, and some input and output videos, is currently stored in /data/donaldson/ants/ant_tracker. To execute the code, navigate to that directory and run ./pipeline.py <input-files>, where <input-files> is a list of all the files which you wish to process. Optionally, the --labels or --edges options may be specified to only create the labels or the edges for the ROIs, respectively. Otherwise, it will produce both for all input files. If you don't like the ROIs that were detected, you can create your own using the roidefine.py script.
You should run the code in a screen session to enable it to continue running in case your session ends, and you should give it a positive nice value to allow other users of the computer to work with the system. To handle both of those, run nice -n 4 screen, then execute the snakemake commands in the screen session. You can then press Ctrl+a, then d to detach the session from your terminal to do other things on purves, run screen -r from within your terminal to reattach it.
To upload videos to purves, and to download the results from the pipeline, I recommend using scp.
If some error occurs in the execution of the pipeline, then you need to delete all of the files which it writes into the intermediate directory for the video (the files should be the intermediate/split/<video-name>, intermediate/crop/<video-name>, intermediate/track/<video-name>, and intermediate/aggregate/<video-name> directories, and the intermediate/rois/<video-name>.txt file.
For downloading the videos onto purves, I recommend using the gddown.py script which I wrote to easily specify mass downloads.
If you want to execute this code on another machine, then you'll need to clone this repository onto the computer you want to run it on. Once you've done that, install all the required dependencies (the check-dependencies.py script will be very helpful here), and then make a folder named input/ in the root of this repository. Since the pipeline is not being run on purves, the virtual environment is not required to run the pipeline. If you are running on a Windows machine, it's recommended you also install WSL (Windows Subsystem for Linux) to resolve any potential pathing issues.
After those two steps, you should be able to execute the pipeline using the same commands as you would on purves.
Most of the workflow will be the same, but you can tell Snakemake to execute on a cluster. The instructions are here, and you can figure that out if you want.