Centrifuge error while using reference mode

[ekiefl]

I got this error running in reference mode and I'm not sure how to diagnose.

rule run_centrifuge:
    input: 03_CONTIGS/G01-gene-calls.fa
    output: 03_CONTIGS/G01-centrifuge_hits.tsv, 03_CONTIGS/G01-centrifuge_report.tsv
    log: 00_LOGS/G01-run_centrifuge.log
    jobid: 5986
    wildcards: group=G01
    threads: 5
    resources: nodes=5

centrifuge -f -x $CENTRIFUGE_BASE/p+h+v/p+h+v 03_CONTIGS/G01-gene-calls.fa -S 03_CONTIGS/G01-centrifuge_hits.tsv --report-file 03_CONTIGS/G01-centrifuge_report.tsv --threads 5 >> 00_LOGS/G01-run_centrifuge.log 2>&1
Finished job 13414.
7044 of 16441 steps (43%) done
Finished job 14371.
7045 of 16441 steps (43%) done
Finished job 13413.
7046 of 16441 steps (43%) done
Finished job 1021.
7047 of 16441 steps (43%) done
Error in job run_centrifuge while creating output files 03_CONTIGS/G01-centrifuge_hits.tsv, 03_CONTIGS/G01-centrifuge_report.tsv.
ClusterJobException in line 632 of /groups/merenlab/people/ekiefl/utils/MerenLab-workflows/workflows/assembly-based-metagenomics-workflow/merenlab-metagenomics-pipeline.snakefile:
Error executing rule run_centrifuge on cluster (jobid: 5986, external: Your job 673974 ("snakejob.run_centrifuge.5986.sh") has been submitted, jobscript: /storage/data/00_Ecoli_Evolution_Metagenomes/.snakemake/tmp.3yv7dxit/snakejob.run_centrifuge.5986.sh). For detailed error see the cluster log.
Will exit after finishing currently running jobs.
Touching output file 03_CONTIGS/anvi_run_ncbi_cogs-G01.done.
Finished job 2075.
7048 of 16441 steps (43%) done
Will exit after finishing currently running jobs.
Finished job 10567.
7049 of 16441 steps (43%) done
Will exit after finishing currently running jobs.
Finished job 10842.
7050 of 16441 steps (43%) done
Will exit after finishing currently running jobs.
Finished job 9583.
7051 of 16441 steps (43%) done
Will exit after finishing currently running jobs.
Exiting because a job execution failed. Look above for error message

This is from 00_LOGS:

/storage/data/00_Ecoli_Evolution_Metagenomes/iernECsMaF.sh
command line: perl /bioware/seqinfo/bin/clusterize -n 5 -log 00_LOGS/G01-run_centrifuge.log /storage/data/00_Ecoli_Evolution_Metagenomes/.snakemake/tmp.3yv7dxit/snakejob.run_centrifuge.5986.sh
Running on host: barhal-01.bpcservers.private
Running command: /storage/data/00_Ecoli_Evolution_Metagenomes/.snakemake/tmp.3yv7dxit/snakejob.run_centrifuge.5986.sh
centrifuge -f -x $CENTRIFUGE_BASE/p+h+v/p+h+v 03_CONTIGS/G01-gene-calls.fa -S 03_CONTIGS/G01-centrifuge_hits.tsv --report-file 03_CONTIGS/G01-centrifuge_report.tsv --threads 5 >> 00_LOGS/G01-run_centrifuge.log 2>&1
/bin/bash: CENTRIFUGE_BASE: unbound variable
Error in job run_centrifuge while creating output files 03_CONTIGS/G01-centrifuge_report.tsv, 03_CONTIGS/G01-centrifuge_hits.tsv.
RuleException:
CalledProcessError in line 643 of /groups/merenlab/people/ekiefl/utils/MerenLab-workflows/workflows/assembly-based-metagenomics-workflow/merenlab-metagenomics-pipeline.snakefile:
Command 'centrifuge -f -x $CENTRIFUGE_BASE/p+h+v/p+h+v 03_CONTIGS/G01-gene-calls.fa -S 03_CONTIGS/G01-centrifuge_hits.tsv --report-file 03_CONTIGS/G01-centrifuge_report.tsv --threads 5 >> 00_LOGS/G01-run_centrifuge.log 2>&1' returned non-zero exit status 1
  File "/groups/merenlab/people/ekiefl/utils/MerenLab-workflows/workflows/assembly-based-metagenomics-workflow/merenlab-metagenomics-pipeline.snakefile", line 643, in __rule_run_centrifuge
  File "/bioware/python-3.5.1-201606221454/lib/python3.5/concurrent/futures/thread.py", line 55, in run
Exiting because a job execution failed. Look above for error message
Job exit status: 1
Warning: job exited non-zero: 1

One problem with snakemake is that the error codes are not very helpful.

[meren]

Do you think "/bin/bash: CENTRIFUGE_BASE: unbound variable" has anything to do with it? Can you run `echo $CENTRIFUGE_BASE` on your term?

…

-- A. Murat Eren (meren) http://merenlab.org :: twitter <https://twitter.com/merenbey> :: gpg <https://keybase.io/meren>

On Sun, Jan 14, 2018 at 12:38 PM, ekiefl ***@***.***> wrote: I got this error running in reference mode and I'm not sure how to diagnose. rule run_centrifuge: input: 03_CONTIGS/G01-gene-calls.fa output: 03_CONTIGS/G01-centrifuge_hits.tsv, 03_CONTIGS/G01-centrifuge_report.tsv log: 00_LOGS/G01-run_centrifuge.log jobid: 5986 wildcards: group=G01 threads: 5 resources: nodes=5 centrifuge -f -x $CENTRIFUGE_BASE/p+h+v/p+h+v 03_CONTIGS/G01-gene-calls.fa -S 03_CONTIGS/G01-centrifuge_hits.tsv --report-file 03_CONTIGS/G01-centrifuge_report.tsv --threads 5 >> 00_LOGS/G01-run_centrifuge.log 2>&1 Finished job 13414. 7044 of 16441 steps (43%) done Finished job 14371. 7045 of 16441 steps (43%) done Finished job 13413. 7046 of 16441 steps (43%) done Finished job 1021. 7047 of 16441 steps (43%) done Error in job run_centrifuge while creating output files 03_CONTIGS/G01-centrifuge_hits.tsv, 03_CONTIGS/G01-centrifuge_report.tsv. ClusterJobException in line 632 of /groups/merenlab/people/ekiefl/utils/MerenLab-workflows/workflows/assembly-based-metagenomics-workflow/merenlab-metagenomics-pipeline.snakefile: Error executing rule run_centrifuge on cluster (jobid: 5986, external: Your job 673974 ("snakejob.run_centrifuge.5986.sh") has been submitted, jobscript: /storage/data/00_Ecoli_Evolution_Metagenomes/.snakemake/tmp.3yv7dxit/snakejob.run_centrifuge.5986.sh). For detailed error see the cluster log. Will exit after finishing currently running jobs. Touching output file 03_CONTIGS/anvi_run_ncbi_cogs-G01.done. Finished job 2075. 7048 of 16441 steps (43%) done Will exit after finishing currently running jobs. Finished job 10567. 7049 of 16441 steps (43%) done Will exit after finishing currently running jobs. Finished job 10842. 7050 of 16441 steps (43%) done Will exit after finishing currently running jobs. Finished job 9583. 7051 of 16441 steps (43%) done Will exit after finishing currently running jobs. Exiting because a job execution failed. Look above for error message This is from 00_LOGS: /storage/data/00_Ecoli_Evolution_Metagenomes/iernECsMaF.sh command line: perl /bioware/seqinfo/bin/clusterize -n 5 -log 00_LOGS/G01-run_centrifuge.log /storage/data/00_Ecoli_Evolution_Metagenomes/.snakemake/tmp.3yv7dxit/snakejob.run_centrifuge.5986.sh Running on host: barhal-01.bpcservers.private Running command: /storage/data/00_Ecoli_Evolution_Metagenomes/.snakemake/tmp.3yv7dxit/snakejob.run_centrifuge.5986.sh centrifuge -f -x $CENTRIFUGE_BASE/p+h+v/p+h+v 03_CONTIGS/G01-gene-calls.fa -S 03_CONTIGS/G01-centrifuge_hits.tsv --report-file 03_CONTIGS/G01-centrifuge_report.tsv --threads 5 >> 00_LOGS/G01-run_centrifuge.log 2>&1 /bin/bash: CENTRIFUGE_BASE: unbound variable Error in job run_centrifuge while creating output files 03_CONTIGS/G01-centrifuge_report.tsv, 03_CONTIGS/G01-centrifuge_hits.tsv. RuleException: CalledProcessError in line 643 of /groups/merenlab/people/ekiefl/utils/MerenLab-workflows/workflows/assembly-based-metagenomics-workflow/merenlab-metagenomics-pipeline.snakefile: Command 'centrifuge -f -x $CENTRIFUGE_BASE/p+h+v/p+h+v 03_CONTIGS/G01-gene-calls.fa -S 03_CONTIGS/G01-centrifuge_hits.tsv --report-file 03_CONTIGS/G01-centrifuge_report.tsv --threads 5 >> 00_LOGS/G01-run_centrifuge.log 2>&1' returned non-zero exit status 1 File "/groups/merenlab/people/ekiefl/utils/MerenLab-workflows/workflows/assembly-based-metagenomics-workflow/merenlab-metagenomics-pipeline.snakefile", line 643, in __rule_run_centrifuge File "/bioware/python-3.5.1-201606221454/lib/python3.5/concurrent/futures/thread.py", line 55, in run Exiting because a job execution failed. Look above for error message Job exit status: 1 Warning: job exited non-zero: 1 One problem with snakemake is that the error codes are not very helpful. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#13>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMCu6yPcIgeVMriMk9YAgvxbrY9DMhTks5tKkmygaJpZM4RdsHm> .

[ShaiberAlon]

Meren is right, I wrote the solution on the github slack channel. On Sun, Jan 14, 2018 at 1:07 PM A. Murat Eren <[email protected]> wrote:

…

Do you think "/bin/bash: CENTRIFUGE_BASE: unbound variable" has anything to do with it? Can you run `echo $CENTRIFUGE_BASE` on your term? -- A. Murat Eren (meren) http://merenlab.org :: twitter <https://twitter.com/merenbey> :: gpg <https://keybase.io/meren> On Sun, Jan 14, 2018 at 12:38 PM, ekiefl ***@***.***> wrote: > I got this error running in reference mode and I'm not sure how to > diagnose. > > rule run_centrifuge: > input: 03_CONTIGS/G01-gene-calls.fa > output: 03_CONTIGS/G01-centrifuge_hits.tsv, 03_CONTIGS/G01-centrifuge_report.tsv > log: 00_LOGS/G01-run_centrifuge.log > jobid: 5986 > wildcards: group=G01 > threads: 5 > resources: nodes=5 > > centrifuge -f -x $CENTRIFUGE_BASE/p+h+v/p+h+v 03_CONTIGS/G01-gene-calls.fa -S 03_CONTIGS/G01-centrifuge_hits.tsv --report-file 03_CONTIGS/G01-centrifuge_report.tsv --threads 5 >> 00_LOGS/G01-run_centrifuge.log 2>&1 > Finished job 13414. > 7044 of 16441 steps (43%) done > Finished job 14371. > 7045 of 16441 steps (43%) done > Finished job 13413. > 7046 of 16441 steps (43%) done > Finished job 1021. > 7047 of 16441 steps (43%) done > Error in job run_centrifuge while creating output files 03_CONTIGS/G01-centrifuge_hits.tsv, 03_CONTIGS/G01-centrifuge_report.tsv. > ClusterJobException in line 632 of /groups/merenlab/people/ekiefl/utils/MerenLab-workflows/workflows/assembly-based-metagenomics-workflow/merenlab-metagenomics-pipeline.snakefile: > Error executing rule run_centrifuge on cluster (jobid: 5986, external: Your job 673974 ("snakejob.run_centrifuge.5986.sh") has been submitted, jobscript: /storage/data/00_Ecoli_Evolution_Metagenomes/.snakemake/tmp.3yv7dxit/ snakejob.run_centrifuge.5986.sh). For detailed error see the cluster log. > Will exit after finishing currently running jobs. > Touching output file 03_CONTIGS/anvi_run_ncbi_cogs-G01.done. > Finished job 2075. > 7048 of 16441 steps (43%) done > Will exit after finishing currently running jobs. > Finished job 10567. > 7049 of 16441 steps (43%) done > Will exit after finishing currently running jobs. > Finished job 10842. > 7050 of 16441 steps (43%) done > Will exit after finishing currently running jobs. > Finished job 9583. > 7051 of 16441 steps (43%) done > Will exit after finishing currently running jobs. > Exiting because a job execution failed. Look above for error message > > This is from 00_LOGS: > > /storage/data/00_Ecoli_Evolution_Metagenomes/iernECsMaF.sh > command line: perl /bioware/seqinfo/bin/clusterize -n 5 -log 00_LOGS/G01-run_centrifuge.log /storage/data/00_Ecoli_Evolution_Metagenomes/.snakemake/tmp.3yv7dxit/ snakejob.run_centrifuge.5986.sh > Running on host: barhal-01.bpcservers.private > Running command: /storage/data/00_Ecoli_Evolution_Metagenomes/.snakemake/tmp.3yv7dxit/ snakejob.run_centrifuge.5986.sh > centrifuge -f -x $CENTRIFUGE_BASE/p+h+v/p+h+v 03_CONTIGS/G01-gene-calls.fa -S 03_CONTIGS/G01-centrifuge_hits.tsv --report-file 03_CONTIGS/G01-centrifuge_report.tsv --threads 5 >> 00_LOGS/G01-run_centrifuge.log 2>&1 > /bin/bash: CENTRIFUGE_BASE: unbound variable > Error in job run_centrifuge while creating output files 03_CONTIGS/G01-centrifuge_report.tsv, 03_CONTIGS/G01-centrifuge_hits.tsv. > RuleException: > CalledProcessError in line 643 of /groups/merenlab/people/ekiefl/utils/MerenLab-workflows/workflows/assembly-based-metagenomics-workflow/merenlab-metagenomics-pipeline.snakefile: > Command 'centrifuge -f -x $CENTRIFUGE_BASE/p+h+v/p+h+v 03_CONTIGS/G01-gene-calls.fa -S 03_CONTIGS/G01-centrifuge_hits.tsv --report-file 03_CONTIGS/G01-centrifuge_report.tsv --threads 5 >> 00_LOGS/G01-run_centrifuge.log 2>&1' returned non-zero exit status 1 > File "/groups/merenlab/people/ekiefl/utils/MerenLab-workflows/workflows/assembly-based-metagenomics-workflow/merenlab-metagenomics-pipeline.snakefile", line 643, in __rule_run_centrifuge > File "/bioware/python-3.5.1-201606221454/lib/python3.5/concurrent/futures/thread.py", line 55, in run > Exiting because a job execution failed. Look above for error message > Job exit status: 1 > Warning: job exited non-zero: 1 > > One problem with snakemake is that the error codes are not very helpful. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#13>, or mute the > thread > < https://github.com/notifications/unsubscribe-auth/AAMCu6yPcIgeVMriMk9YAgvxbrY9DMhTks5tKkmygaJpZM4RdsHm > > . > — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#13 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQ18dPJ3e3d3esHEkoXXzF-0OCU3Qocnks5tKlBmgaJpZM4RdsHm> .