A general metagenomics data mining system focus on robust microbiome research.
metapi works with Python 3.7+. You can install it via bioconda:
# It is recommended to install the latest version
➤ mamba install -c conda-forge -c bioconda metapi=3.0.0
or install latest version using git:
➤ mamba create -n metapi-env python=3.10 snakemake fd-find seqkit seqtk ruamel.yaml pandas numpy natsort openpyxl biopython seaborn matplotlib executor
➤ git clone https://github.com/ohmeta/metapi
➤ echo "export PYTHONPATH=/path/to/metapi:$PYTHONPATH" >> ~/.bashrc # if you use bash
➤ echo "export PYTHONPATH=/path/to/metapi:$PYTHONPATH" >> ~/.zshrc # if you use zsh
➤ echo "set -x PYTHONPATH /path/to/metapi $PYTHONPATH" >> ~/.config/fish/config.fish # if you use fish
➤ mamba activate metapi-env
➤ python /path/to/metapi/run_metapi.py --help # using run_metapi.py instead of metapi
➤ metapi --help
.___ ___. _______ .___________. ___ .______ __
| \/ | | ____|| | / \ | _ \ | |
| \ / | | |__ `---| |----` / ^ \ | |_) | | |
| |\/| | | __| | | / /_\ \ | ___/ | |
| | | | | |____ | | / _____ \ | | | |
|__| |__| |_______| |__| /__/ \__\ | _| |__|
Omics for All, Open Source for All
A general metagenomics data mining system focus on robust microbiome research.
optional arguments:
-h, --help show this help message and exit
-v, --version print software version and exit
available subcommands:
init init project
simulate_wf simulate reads
mag_wf metagenome-assembly-genome pipeline
gene_wf metagenome-assembly-gene pipeline
sync metapi sync project
➤ metapi init --help
usage: metapi init [-h] [-d WORKDIR] [--check-samples] [-s SAMPLES]
[-b {simulate,trimming,rmhost,assembly,binning,checkm}]
[--trimmer {sickle,fastp,trimmomatic}]
[--rmhoster {bwa,bowtie2,minimap2,kraken2,kneaddata}]
[--assembler {idba-ud,megahit,metaspades,spades,opera-ms} [{idba-ud,megahit,metaspades,spades,opera-ms} ...]]
[--binner BINNER [BINNER ...]]
optional arguments:
-h, --help show this help message and exit
-d, --workdir WORKDIR
project workdir (default: ./)
--check-samples check samples, default: False
-s, --samples SAMPLES
desired input:
samples list, tsv format required.
if begin from trimming, rmhost, or assembly:
if it is fastq:
the header is: [sample_id, assembly_group, binning_group, fq1, fq2]
if it is sra:
the header is: [sample_id, assembly_group, binning_group, sra]
if begin from simulate:
the header is: [id, genome, abundance, reads_num, model]
-b, --begin {simulate,trimming,rmhost,assembly,binning,checkm}
pipeline starting point (default: trimming)
--trimmer {sickle,fastp,trimmomatic}
which trimmer used (default: fastp)
--rmhoster {bwa,bowtie2,minimap2,kraken2,kneaddata}
which rmhoster used (default: bowtie2)
--assembler {idba-ud,megahit,metaspades,spades,opera-ms} [{idba-ud,megahit,metaspades,spades,opera-ms} ...]
which assembler used, required when begin with binning, can be changed in config.yaml (default: ['metaspades'])
--binner BINNER [BINNER ...]
wchich binner used (default: ['metabat2', 'concoct', 'maxbin2', 'vamb', 'dastools'])
- Note
-
When we do
metapi init, metapi will help us to create project structure, includeconfig.yaml,profiles/,envs/,logs/andresults/.config.yaml: workflow configuration, can be editedprofiles/: used when run pipeline on cluster, can be edited- For
SGEuser, please seeprofiles/sge/cluster.yaml - For
SLURMuser, please seeprofiles/slurm/cluster.yaml
- For
envs/: conda environment file for some rules, can be edited to update software version by yourself
-
We recommand you to create a directory, and
cdit and putsamples.tsvin the directory, then dometapi init -s samples.tsv -d ./. -
Samples file is required when init project. Samples input format requirement can see here.
-
Begin point can be
simulate,trimming,rmhost,assemblyandbinning, it means:- simulate: simulate reads
- trimming: samples need trimming, do quality control
- rmhost: samples is quality control well, only need to remove host sequence, will not do trimming
- assembly: samples is clean, just do assembly, will not do trimming and rmhost
- (WIP) binning: supply samples and assembly, then do binning, will no do trimming, rmhost and assembly
-
When metapi init executed, then corresponding configuration will be writen into
config.yaml. Of course, you can editconfig.yamlto update config, then when runmetapi mag_wformetapi gene_wf,metapiwill understand it. Just edit it, see what will happen.
-
➤ metapi mag_wf --help
[--cluster-engine {slurm,sge,lsf,pbs-torque}]
[--wait WAIT] [--use-conda] [--conda-prefix CONDA_PREFIX]
[--conda-create-envs-only]
[TASK]
positional arguments:
TASK pipeline end point. Allowed values are
raw_prepare_reads_all,
raw_fastqc_all,
raw_report_all,
raw_all,
trimming_sickle_all,
trimming_fastp_all,
trimming_trimmomatic_all,
trimming_report_all,
trimming_all,
rmhost_bwa_all,
rmhost_bowtie2_all,
rmhost_minimap2_all,
rmhost_kraken2_all,
rmhost_kneaddata_all,
rmhost_report_all,
rmhost_all,
qcreport_all,
assembly_megahit_all,
assembly_idba_ud_all,
assembly_metaspades_all,
assembly_spades_all,
assembly_plass_all,
assembly_opera_ms_all,
assembly_metaquast_all,
assembly_report_all,
assembly_all,
alignment_base_depth_all,
alignment_report_all,
alignment_all,
binning_metabat2_coverage_all,
binning_metabat2_all,
binning_maxbin2_all,
binning_concoct_all,
binning_graphbin2_all,
binning_dastools_all,
binning_vamb_prepare_all,
binning_vamb_all,
binning_report_all,
binning_all,
identify_virsorter2_all,
identify_deepvirfinder_all,
identify_single_all,
identify_phamb_all,
identify_multi_all,
identify_all,
predict_scaftigs_gene_prodigal_all,
predict_scaftigs_gene_prokka_all,
predict_mags_gene_prodigal_all,
predict_mags_gene_prokka_all,
predict_scaftigs_gene_all,
predict_mags_gene_all,
predict_all,
annotation_prophage_dbscan_swa_all,
annotation_all,
checkm_all,
checkv_all,
check_all,
dereplicate_mags_drep_all,
dereplicate_mags_galah_all,
dereplicate_mags_all,
dereplicate_vmags_all,
dereplicate_all,
taxonomic_gtdbtk_all,
taxonomic_all,
databases_bacteriome_kmcp_all,
databases_bacteriome_kraken2_all,
databases_bacteriome_krakenuniq_all,
databases_bacteriome_all,
databases_all,
upload_sequencing_all,
upload_assembly_all,
upload_all,
all (default: all)
optional arguments:
-h, --help show this help message and exit
-d, --workdir WORKDIR
project workdir (default: ./)
--check-samples check samples, default: False
--config CONFIG config.yaml (default: ./config.yaml)
--profile PROFILE cluster profile name (default: ./profiles/slurm)
--cores CORES all job cores, available on '--run-local' (default: 240)
--local-cores LOCAL_CORES
local job cores, available on '--run-remote' (default: 8)
--jobs JOBS cluster job numbers, available on '--run-remote' (default: 30)
--list list pipeline rules
--debug debug pipeline
--dry-run dry run pipeline
--run-local run pipeline on local computer
--run-remote run pipeline on remote cluster
--cluster-engine {slurm,sge,lsf,pbs-torque}
cluster workflow manager engine, support slurm(sbatch) and sge(qsub) (default: slurm)
--wait WAIT wait given seconds (default: 60)
--use-conda use conda environment
--conda-prefix CONDA_PREFIX
conda environment prefix (default: ~/.conda/envs)
--conda-create-envs-only
conda create environments only
# init project
➤ metapi init -d . -s samples.tsv -b trimming
# create conda environments, which will create envs at ~/.conda/envs
# run it firstly, recommend
➤ metapi mag_wf --conda-create-envs-only
# run pipeline with conda
➤ metapi mag_wf all --use-conda --run-local
# run raw_fastqc
➤ metapi mag_wf raw_fastqc_all --use-conda --run-local
# run trimming
➤ metapi mag_wf trimming_all --use-conda --run-local
# run rmhost
➤ metapi mag_wf rmhost_all --use-conda --run-local
# run qc report
➤ metapi mag_wf qcreport_all --use-conda --run-local
# run assembly
➤ metapi mag_wf assembly_all --use-conda --run-local
# run binning (microbial MAGs)
➤ metapi mag_wf binning_all --use-conda --run-local
# run identify (viral MAGs)
➤ metapi mag_wf identify_all --use-conda --run-local
# run gene predict
➤ metapi mag_wf predict_all --use-conda --run-local
# run checkm on microbial MAGs
➤ metapi mag_wf checkm_all --use-conda --run-local
# run checkv on viral MAGs
➤ metapi mag_wf checkv_all --use-conda --run-local
# run dereplicate on microbial MAGs
➤ metapi mag_wf dereplicate_mags_all --use-conda --run-local
# run dereplicate on viral MAGs
➤ metapi mag_wf dereplicate_vmags_all --use-conda --run-local
#run taxonomic (currently on microbial MAGs)
➤ metapi mag_wf taxonomic_all --use-conda --run-local
# run databases (generate databases for profiling)
➤ metapi mag_wf databases_all --use-conda --run-local
# run mag_wf all
➤ metapi mag_wf --use-conda --run-local
# run mag_wf all on SGE/SLURM cluster
➤ metapi mag_wf all --use-conda --run-remote
-
The input fastq need gzip compress.
-
If
sample_idcol contain same id, then the reads of each sample will be merged. -
For the same
assembly_group, the reads will be combined together to do co-assembly, then the reads of each sample will map to the co-assembled results to calculate co-abundance for eachassembly_group, then using MetaBAT2, MaxBin2, CONCOCT (You can specific which binner to use) to do binning. -
For the same
binning_group, the assembled results will be combined together to build minimap2 index, then the reads of each sample will map to the combined index to calculate co-abundance, then using vamb to do binning -
assembly_groupcontrol the assembly and binning strategy;binning_groupcontrol the multisplit binning strategy developed by vamb. -
If all
assembly_groupare unique, for MetaBAT2, MaxBin2, CONCOCT, etc., that is the traditional single-assembly and single-binning strategy. -
If
binning_grouphas multiple groups, then vamb will perform multisplit binning in eachbinning_groupseparately, which is useful for very large-scale data. For example, if there are 10,000 samples, and multisplit binning is performed once for every thousand samples, the binning group can be set to 10 different groups.
-
begin from trimming, rmhost or assembly:
Paired-end reads
sample_id assembly_group binning_group short_forward_reads short_reverse_reads s1 ag1 bg1 s1.1.fq.gz s1.2.fq.gz s2 ag1 bg1 s2.1.fq.gz s2.2.fq.gz s3 ag2 bg2 s3.1.fq.gz s3.2.fq.gz s4 ag2 bg2 s4.1.fq.gz s4.2.fq.gz Paired-end reads (interleaved)
sample_id assembly_group binning_group short_interleaved_reads s1 ag1 bg1 s1.fq.gz s2 ag1 bg1 s2.fq.gz s3 ag2 bg2 s3.fq.gz s4 ag2 bg2 s4.fq.gz Single-end reads
sample_id assembly_group binning_group short_single_reads s1 ag1 bg1 s1.fq.gz s2 ag1 bg1 s2.fq.gz s3 ag2 bg2 s3.fq.gz s4 ag2 bg2 s4.fq.gz - (WIP)
Paired-end reads and/or single reads
sample_id assembly_group binning_group short_forward_reads short_reverse_reads short_single_reads s1 ag1 bg1 s1.1.fq.gz s1.2.fq.gz s1.fq.gz s2 ag1 bg1 s2.1.fq.gz s2.2.fq.gz s2.fq.gz s3 ag2 bg2 s3.1.fq.gz s3.2.fq.gz s3.fq.gz s4 ag2 bg2 s4.1.fq.gz s4.2.fq.gz s4.fq.gz Paired-end reads and/or single reads, and/or long reads
sample_id assembly_group binning_group short_forward_reads short_reverse_reads short_single_reads long_reads s1 ag1 bg1 s1.1.fq.gz s1.2.fq.gz s1.fq.gz s1.long.fq.gz s2 ag1 bg1 s2.1.fq.gz s2.2.fq.gz s2.fq.gz s2.long.fq.gz s3 ag2 bg2 s3.1.fq.gz s3.2.fq.gz s3.fq.gz s3.long.fq.gz s4 ag2 bg2 s4.1.fq.gz s4.2.fq.gz s4.fq.gz s4.long.fq.gz SRA PE and/or SE: SRA can be dumpped to Paired-end/Single-end fastq reads
sample_id assembly_group binning_group sra_pe sra_se s1 ag1 bg1 s1.pe.sra s1.se.sra s2 ag1 bg1 s2.pe.sra s2.se.sra s3 ag2 bg2 s3.pe.sra s3.se.sra s4 ag2 bg2 s4.pe.sra s4.se.sra -
(WIP) Begin from simulate, only support paired-end reads
id genome abundance reads_num model s1 g1.fa 1.0 1M hiseq s2 g1.fa 0.5 2M hiseq s2 g2.fa 0.5 2M hiseq s3 g1.fa 0.2 3M hiseq s3 g2.fa 0.3 3M hiseq s3 g3.fa 0.5 3M hiseq
It means:
The sample s1 contain 1M reads which come from g1, the relative abundance of species g1 is 1.0.
The sample s2 contain 2M reads, 1M reads come from g1 and 1M reads come from g2. the relative abundance of species g1 is 0.5, the relative abundance of species g2 is 0.5.
The sample s3 contain 3M reads, 0.6M reads come from g1, 0.9M reads come from g2 and 1.5M reads come from g3, the relative abundance of species g1 is 0.2, the relative abundance of species g2 is 0.3, the relative abundance of species g3 is 0.5.
Then metapi will use InSilicoSeq to generate metagenomics shotgun reads.
-
(WIP) begin from binning
sample_id assembly_group binning_group fq1 fq2 scaftigs s1 ag1 bg1 s1.1.fq.gz s1.2.fq.gz ag1.scaftigs.fa.gz s2 ag1 bg1 s2.1.fq.gz s2.2.fq.gz ag1.scaftigs.fa.gz s3 ag2 bg2 s3.1.fq.gz s3.2.fq.gz ag2.scaftigs.fa.gz s4 ag2 bg2 s4.1.fq.gz s4.2.fq.gz ag2.scaftigs.fa.gz -
(WIP) begin from checkm, or dereplicate, or taxonomic assignment (fq1 and fq2 can be empty)
sample_id bins s1 s1.bin.1.fa s1 s1.bin.2.fa s1 s1.bin.3.fa s2 s2.bin.1.fa s2 s2.bin.2.fa s2 s3.bin.3.fa
- config.yaml
- logs/
- profiles/
- envs/
- results/
00.raw/
01.trimming/
02.rmhost/
03.qcreport/
04.assembly/
05.alignment/
06.binning/
06.identify/
07.annotation/
07.predict/
08.check/
09.dereplicate/
10.taxonomic/
11.databases/
99.upload/
- We will try our best to keep the directory structure uniform. Sequence files are generally placed in the reads directory, and report files are generally placed in the report directory.
- If you are not very clear about the output of the whole process, it is recommended to directly raise an issue or look at the code.
- miMompletedb
- VOGdb: untar vog.hmm.tar.gz to get all hmm files. Concatenate all the hmm-files these into an AllVOG.hmm file.
- randomforest_script
If you want to report a bug or issue, or have problems with installing or running the software, please create a new issue. This is the preferred way of getting support. Alternatively, you can mail me.
Contributions welcome! Send me a pull request or get in touch.
When contributing a PR, please use the dev branch.
For style, code will be checked using flake8,
black, and
snakefmt. These modules can be
installed via conda, conda install black flake8 flake8-bugbear snakefmt or via
pip pip install black flake8 flake8-bugbear snakefmt.
- Jie Zhu - @alienzj
- Fangming Yang - @yangfangming
- Yanmei Ju - @juyanmei
- Weiting Liang - @weiting-liang
- Ye Peng - @peng-ye
Over 50,000 Metagenomically Assembled Draft Genomes for the Human Oral Microbiome Reveal New Taxa
Genomics, Proteomics & Bioinformatics, 2021, https://doi.org/10.1016/j.gpb.2021.05.001
This module is licensed under the terms of the GPLv3 license.