SLURM Wrappers

❗ This repository is under construction. ❗

A set of SLURM-compliant codebase wrappers originally designed to interact with the Minnesota Supercomputing Institute (MSI). However, the wrappers should be able to function on all systems that use SLURM and have the codebase requirements installed.

Most of these wrappers are tailored toward working with the S3 via s3cmd or boto3, but some are designed to work on disk and all can be adapted to work on disk. Using the S3, especially on MSI, is highly encouraged.

Cloning a Single Wrapper

To clone a certain wrapper (subdirectory), you can use git sparse checkout. If you have a version of Git below 2.25.0 use these commands:

mkdir <repo>
cd <repo>
git init
git remote add -f origin <repo-url>
git config core.sparseCheckout true
echo "some/dir/" >> .git/info/sparse-checkout # Add folder to checkout
echo "another/dir/" >> .git/info/sparse-checkout
cat .git/info/sparse-checkout # Check which folders you've added
git pull origin master

If you have a version of git above 2.25.0, you can use these commands instead:

git sparse-checkout init   # Same as git config core.sparseCheckout true
git sparse-checkout set "some/dir"   # Same as echo "some/dir/" >> .git/info/sparse-checkout
git sparse-checkout list   # Same as cat .git/info/sparse-checkout

Requirements

It is recommended that you go to the GitHub repository for the codebase itself in order to make sure all of the requirements are satisfied.

Wrapper Descriptions

• ABCD-related: ABCD-BIDS pipeline, abcd-dicom2bids, and ABCD-BIDS + file-mapper + DEAP derivative generation job submission.
• BIBSnet: BIBSnet single echo tier 1 and s3 job submission.
• CABINET: CABINET tier 1 and s3 job submission.
• call_R: TBD
• continuous_submitter: submission of any jobs at a given rate to pace queueing.
• CuBIDS: CuBIDS s3 job submission (appending outputs).
• CustomClean: standalone (outside of ABCD-BIDS) CustomClean s3 job submission.
• DEAP-derivatives: DEAP derivative generation s3 job submission.
• disk_usage: disk usage and file count s3 job submission.
• file-mapper: file-mapper s3 job submission.
• fMRIprep: fMRIprep s3 job submission.
• HCPYA_BIDS_convert: HCP-YA BIDS conversion s3 job submission.
• infant: BCP DCAN-infant, BIBSnet + DCAN-infant, general DCAN-infant, nibabies, and nibabies + XCP-D s3 job submission.
• infomap: infomap (LINK TBD) s3 job submission.
• intendedfors: IntendedFor-matching s3 job submission.
• motion_tsv_generation: s3 job submission for generating motion TSV files from ABCD-BIDS 0.0.3 outputs.
• move_subjects: s3 job submission for moving files from one bucket to another.
• QSIprep: QSIprep s3 job submission.
• s3_sync_external: s3 job submission for syncing from one s3 host to another, utilizing 2 s3 configs.
• s3_system_transfer_subset: s3 job submission for syncing a file subset from one s3 host to another, utilizing 2 s3 configs.
• simnibs_cifti_tools: TBD