feature/bmeg

[bwalsh]

This PR:

• adds bmeg extract config & tests
• still a WIP

• need to add schema for yaml files - see tests/fixtures/bmeg.yaml
• need to test with TES - pending py-tes update
• other? (comments appreciated)

dags/bmeg_pipeline.py will produce this:

tags are used to narrow the scope of the admin UI:

known issues:

For multi step DAGs e.g. ccle extract - the dag generator is inconsistent on when the dataset is updated. Need to discuss - should we enforce that the complexity of this type of pipeline be hidden from the DAG? i.e should we enforce that the extract author provide a single command?

# config snippet
- id: ccle
  extract_commands:
  - curl -L -o source/ccle/CCLE_depMap_19Q1_TPM_transcripts.csv https://depmap.org/portal/download/api/download/external?file_name=ccle%2Fdepmap-rnaseq-expression-data-ccd0.12%2FCCLE_depMap_19Q1_TPM_transcripts.csv
  - python3 transform/ccle/cellline_lookups.py
  - wget https://data.broadinstitute.org/ccle/CCLE_DepMap_18q3_RNAseq_RPKM_20180718.gct
    -O source/ccle/CCLE_DepMap_18q3_RNAseq_RPKM_20180718.gct
  - curl -L -o source/ccle/CCLE_depMap_19Q1_TPM.csv https://depmap.org/portal/download/api/download/external?file_name=ccle%2Fdepmap-rnaseq-expression-data-ccd0.12%2FCCLE_depMap_19Q1_TPM.csv
  - wget https://depmap.org/portal/download/api/download/external?file_name=processed_portal_downloads%2Fdepmap-public-cell-line-metadata-183e.4%2FDepMap-2019q1-celllines_v2.csv
    -O source/ccle/DepMap-2019q1-celllines.csv_v2.csv
  - curl -L -o source/ccle/sample_info.csv https://ndownloader.figshare.com/files/22629137
  - wget https://data.broadinstitute.org/ccle/CCLE_DepMap_18q3_maf_20180718.txt -O
    source/ccle/CCLE_DepMap_18q3_maf_20180718.txt
  outputs:
  - source/ccle/CCLE_depMap_19Q1_TPM_transcripts.csv
  - source/ccle/cellline_id_lookup.tsv
  - source/ccle/cellline_phenotype_lookup.tsv
  - source/ccle/cellline_properties_lookup.tsv
  - source/ccle/CCLE_DepMap_18q3_RNAseq_RPKM_20180718.gct
  - source/ccle/CCLE_depMap_19Q1_TPM.csv
  - source/ccle/DepMap-2019q1-celllines.csv_v2.csv
  - source/ccle/sample_info.csv
  - source/ccle/CCLE_DepMap_18q3_maf_20180718.txt

[bwalsh]

dataset updates now associated w/ last task in multi task dag