Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move processing pipeline to Docker #18

Merged
merged 5 commits into from
Mar 16, 2018
Merged

Move processing pipeline to Docker #18

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a5af5a7
Add Docker - corresponds to jtaroni/multi-plier:v1
jaclyn-taroni Mar 16, 2018
8f75474
Update: ignore Rplots.pdf
jaclyn-taroni Mar 16, 2018
981c9f7
Update: Only check correlation for Affy
jaclyn-taroni Mar 16, 2018
96a85a1
Update: reprocess data in docker container
jaclyn-taroni Mar 16, 2018
70c0ac8
Update: flowchart, new BrainArray version
jaclyn-taroni Mar 16, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,5 @@ recount.all.data.plier.RData

# Ignore hidden metadata files
._*

Rplots\.pdf
Binary file modified NARES/NARES_PC_scatter_with_batch_correct.pdf
View file
Open in desktop
Binary file not shown.
4 changes: 2 additions & 2 deletions NARES/processed/NARES_SCANfast_ComBat.pcl
View file
Open in desktop
Git LFS file not shown
4 changes: 2 additions & 2 deletions NARES/processed/NARES_hugene10st_SCANfast.pcl
View file
Open in desktop
Git LFS file not shown
36 changes: 36 additions & 0 deletions docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
FROM rocker/tidyverse:3.4.3

RUN apt-get update && apt-get install -y \
git \
wget \
curl

RUN R install2.r --error \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this line doing? What is R install2.r? Curious

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The --error flag will make it such that an error is thrown if the package installation fails (instead of a warning) (see: jsta/r-docker-tutorial#41)

--deps TRUE \
foreach \
parallel \
RCurl

# bioconductor packages
RUN R -e "BiocInstaller::biocLite(c('pd.hg.u133.plus.2', 'pd.hg.u133a', 'pd.hg.u133b', 'pd.hg.u95av2', 'pd.hugene.1.0.st.v1', 'pd.hugene.1.1.st.v1', 'SCAN.UPC', 'affy', 'affyio', 'preprocessCore', 'affxparser', 'illuminaHumanv4.db', 'hgug4112a.db', 'biomaRt', 'org.Hs.eg.db', 'qvalue'))"

# brainarray for affymetrix arrays (v22.0.0)
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hgu133plus2hsentrezgprobe_22.0.0.tar.gz')"
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hgu133ahsentrezgprobe_22.0.0.tar.gz')"
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hgu133bhsentrezgprobe_22.0.0.tar.gz')"
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hgu95av2hsentrezgprobe_22.0.0.tar.gz')"
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hugene11sthsentrezgprobe_22.0.0.tar.gz')"
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hugene10sthsentrezgprobe_22.0.0.tar.gz')"
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hgu133plus2hsentrezgcdf_22.0.0.tar.gz')"
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hgu133ahsentrezgcdf_22.0.0.tar.gz')"
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hgu133bhsentrezgcdf_22.0.0.tar.gz')"
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hgu95av2hsentrezgcdf_22.0.0.tar.gz')"
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hugene11sthsentrezgcdf_22.0.0.tar.gz')"
RUN R -e "devtools::install_url('http://mbni.org/customcdf/22.0.0/entrezg.download/hugene10sthsentrezgcdf_22.0.0.tar.gz')"


# packages from github
RUN R -e "devtools::install_github('wgmao/PLIER', ref = 'a2d4a2aa343f9ed4b9b945c04326bebd31533d4d', dependencies = TRUE)"
RUN R -e "devtools::install_github('greenelab/TDM', ref = '0bb5b7e4f2478295c9889aa1bf3cf91f1db11006', dependencies = TRUE)"

RUN R -e "install.packages('doParallel')"
12 changes: 12 additions & 0 deletions docker/list_user_installed_R_packages.R
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# J. Taroni 2018
# Get list of installed R packages on Docker image
# Adapted from:
# https://www.r-bloggers.com/list-of-user-installed-r-packages-and-their-versions/
# https://stackoverflow.com/questions/38481980/get-the-list-of-installed-packages-by-user-in-r

packages.df <- as.data.frame(installed.packages()[, c(1, 3:4)])
packages.df <- packages.df[is.na(packages.df$Priority), 1:2, drop=FALSE]

sink(file.path("docker", "user_installed_R_packages.txt"))
print(packages.df, row.names=FALSE)
sink()
214 changes: 214 additions & 0 deletions docker/user_installed_R_packages.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,214 @@
Package Version
acepack 1.4.1
affxparser 1.50.0
affy 1.56.0
affyio 1.48.0
annotate 1.56.1
AnnotationDbi 1.40.0
assertthat 0.2.0
backports 1.1.2
base64enc 0.1-3
BH 1.66.0-1
bindr 0.1
bindrcpp 0.2
Biobase 2.38.0
BiocGenerics 0.24.0
BiocInstaller 1.28.0
BiocParallel 1.12.0
biomaRt 2.34.2
Biostrings 2.46.0
bit 1.1-12
bit64 0.9-7
bitops 1.0-6
blob 1.1.0
brew 1.0-6
broom 0.4.3
callr 2.0.2
caTools 1.17.1
cellranger 1.1.0
checkmate 1.8.5
cli 1.0.0
colorspace 1.3-2
commonmark 1.4
covr 3.0.1
crayon 1.3.4
curl 3.1
data.table 1.10.4-3
DBI 0.7
dbplyr 1.2.1
debugme 1.1.0
DelayedArray 0.4.1
desc 1.1.1
devtools 1.13.5
dichromat 2.0-0
digest 0.6.15
docopt 0.4.5
doParallel 1.0.11
dplyr 0.7.4
dtplyr 0.0.2
evaluate 0.10.1
feather 0.3.1
ff 2.2-13
forcats 0.3.0
foreach 1.4.4
formatR 1.5
Formula 1.2-2
futile.logger 1.4.3
futile.options 1.0.0
gdata 2.18.0
gdtools 0.1.7
genefilter 1.60.0
GenomeInfoDb 1.14.0
GenomeInfoDbData 1.0.0
GenomicRanges 1.30.3
GEOquery 2.46.15
ggplot2 2.2.1
ggplot2movies 0.0.1
git2r 0.21.0
glmnet 2.0-13
glue 1.2.0
gmailr 0.7.1
gplots 3.0.1
gridExtra 2.3
gtable 0.2.0
gtools 3.5.0
haven 1.1.1
hexbin 1.27.2
hgu133ahsentrezgcdf 22.0.0
hgu133ahsentrezgprobe 22.0.0
hgu133bhsentrezgcdf 22.0.0
hgu133bhsentrezgprobe 22.0.0
hgu133plus2hsentrezgcdf 22.0.0
hgu133plus2hsentrezgprobe 22.0.0
hgu95av2hsentrezgcdf 22.0.0
hgu95av2hsentrezgprobe 22.0.0
hgug4112a.db 3.2.3
highr 0.6
Hmisc 4.1-1
hms 0.4.1
htmlTable 1.11.2
htmltools 0.3.6
htmlwidgets 1.0
httpuv 1.3.6
httr 1.3.1
hugene10sthsentrezgcdf 22.0.0
hugene10sthsentrezgprobe 22.0.0
hugene11sthsentrezgcdf 22.0.0
hugene11sthsentrezgprobe 22.0.0
hunspell 2.9
igraph 1.1.2
illuminaHumanv4.db 1.26.0
IRanges 2.12.0
irlba 2.3.2
iterators 1.0.9
jsonlite 1.5
knitr 1.20
labeling 0.3
Lahman 6.0-0
lambda.r 1.2
latticeExtra 0.6-28
lazyeval 0.2.1
limma 3.34.9
lintr 1.0.2
littler 0.3.3
lubridate 1.7.3
magrittr 1.5
mapproj 1.2-5
maps 3.2.0
maptools 0.9-2
markdown 0.8
MatrixModels 0.4-1
matrixStats 0.53.1
memoise 1.1.0
microbenchmark 1.4-4
mime 0.5
mnormt 1.5-5
mockery 0.4.1
modelr 0.1.1
multcomp 1.4-8
munsell 0.4.3
mvtnorm 1.0-7
nycflights13 0.2.2
oligo 1.42.0
oligoClasses 1.40.0
openssl 1.0
org.Hs.eg.db 3.5.0
pd.hg.u133.plus.2 3.12.0
pd.hg.u133a 3.12.0
pd.hg.u133b 3.12.0
pd.hg.u95av2 3.12.0
pd.hugene.1.0.st.v1 3.14.1
pd.hugene.1.1.st.v1 3.14.1
pheatmap 1.0.8
pillar 1.2.1
pingr 1.1.2
pkgconfig 2.0.1
PLIER 0.99.0
plogr 0.1-1
plyr 1.8.4
praise 1.0.0
preprocessCore 1.40.0
prettyunits 1.0.2
progress 1.1.2
psych 1.7.8
purrr 0.2.4
quantreg 5.35
qvalue 2.10.0
R6 2.2.2
RColorBrewer 1.1-2
Rcpp 0.12.15
RCurl 1.95-4.10
readr 1.1.1
readxl 1.0.0
rematch 1.0.1
remotes 1.1.1
reprex 0.1.2
reshape2 1.4.3
rex 1.1.2
rlang 0.2.0
rmarkdown 1.8
RMySQL 0.10.14
roxygen2 6.0.1
RPostgreSQL 0.6-2
rprojroot 1.3-2
RSQLite 2.0
rstudioapi 0.7
rsvd 0.9
rversions 1.0.3
rvest 0.3.2
S4Vectors 0.16.0
sandwich 2.4-0
scales 0.5.0
SCAN.UPC 2.20.0
selectr 0.3-1
shiny 1.0.5
snow 0.4-2
sourcetools 0.1.6
sp 1.2-7
SparseM 1.77
stringdist 0.9.4.6
stringi 1.1.6
stringr 1.3.0
SummarizedExperiment 1.8.1
sva 3.26.0
svglite 1.2.1
TDM 0.3
testit 0.7
testthat 2.0.0
TH.data 1.0-8
tibble 1.4.2
tidyr 0.8.0
tidyselect 0.2.4
tidyverse 1.2.1
utf8 1.1.3
viridis 0.5.0
viridisLite 0.3.0
whisker 0.3-2
withr 2.1.1
XML 3.98-1.10
xml2 1.2.0
xtable 1.8-2
XVector 0.18.0
yaml 2.1.17
zlibbioc 1.24.0
zoo 1.8-1
Binary file modified isolated-cell-pop/E-MTAB-2452_PCA_scatter_cell_type.pdf
View file
Open in desktop
Binary file not shown.
4 changes: 2 additions & 2 deletions isolated-cell-pop/processed/E-MTAB-2452_hugene11st_SCANfast.pcl
View file
Open in desktop
Git LFS file not shown
2 changes: 2 additions & 0 deletions sle-wb/02-calculate_spearman_affy_norm.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@ CompareSamplesPlotWrapper <- function(processed.dir, png.lead,
function(f) sub(".*\\_", "",
sub("_([^_]*)$", "", f))))

platforms <- platforms[grepl("hgu", platforms)]

# for each platform
for (plt in platforms) {

Expand Down
Binary file modified sle-wb/diagrams/SLE-WB_selected_normalization_flowchart.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified sle-wb/plots/PCA/HGU133PLUS2_RMA_PC1-5_pairs_no.transform.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions sle-wb/plots/PCA/HGU133PLUS2_RMA_PC1-5_pairs_no.transform.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Principal Component cumulative variance explained
PC1 0.719613257054696
PC2 0.778752014167953
PC3 0.818469460904513
PC4 0.840750812307368
PC5 0.859417961468562
PC1 0.719195490028333
PC2 0.778441415090079
PC3 0.818254325770668
PC4 0.840513867481337
PC5 0.859185095462936
Binary file modified sle-wb/plots/PCA/HGU133PLUS2_RMA_PC1-5_pairs_zto.after.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions sle-wb/plots/PCA/HGU133PLUS2_RMA_PC1-5_pairs_zto.after.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Principal Component cumulative variance explained
PC1 0.643553803255725
PC2 0.75185071203084
PC3 0.797812363918736
PC4 0.820004655365133
PC5 0.8382113999406
PC1 0.64331957608224
PC2 0.751744198021767
PC3 0.797718070379041
PC4 0.81987379920512
PC5 0.838091806794775
Binary file modified sle-wb/plots/PCA/HGU133PLUS2_RMA_PC1-5_pairs_zto.before.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions sle-wb/plots/PCA/HGU133PLUS2_RMA_PC1-5_pairs_zto.before.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Principal Component cumulative variance explained
PC1 0.238368049353571
PC2 0.370181339219669
PC3 0.435996104307166
PC4 0.486643920868974
PC5 0.516392902732684
PC1 0.237965511035396
PC2 0.3699417171876
PC3 0.435803905195611
PC4 0.486459091188161
PC5 0.516211496672587
Binary file modified sle-wb/plots/PCA/HGU133PLUS2_SCAN_PC1-5_pairs_no.transform.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions sle-wb/plots/PCA/HGU133PLUS2_SCAN_PC1-5_pairs_no.transform.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Principal Component cumulative variance explained
PC1 0.566067468113633
PC2 0.659907596480561
PC3 0.715506748965104
PC4 0.756431911267807
PC5 0.785072638756046
PC1 0.56636882225646
PC2 0.660153018484501
PC3 0.715676021912042
PC4 0.756585982088227
PC5 0.785198449823178
Binary file modified sle-wb/plots/PCA/HGU133PLUS2_SCAN_PC1-5_pairs_zto.after.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions sle-wb/plots/PCA/HGU133PLUS2_SCAN_PC1-5_pairs_zto.after.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Principal Component cumulative variance explained
PC1 0.535603866969345
PC2 0.651031074010261
PC3 0.688416375086289
PC4 0.721565328849104
PC5 0.738576444935881
PC1 0.535968177687654
PC2 0.651198024224915
PC3 0.688573002223942
PC4 0.721706513584657
PC5 0.738724111447259
Binary file modified sle-wb/plots/PCA/HGU133PLUS2_SCAN_PC1-5_pairs_zto.before.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions sle-wb/plots/PCA/HGU133PLUS2_SCAN_PC1-5_pairs_zto.before.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Principal Component cumulative variance explained
PC1 0.221525601492806
PC2 0.401550674445085
PC3 0.461953932860628
PC4 0.513577212610772
PC5 0.540286690702352
PC1 0.221498959514109
PC2 0.401466795588395
PC3 0.461965892500508
PC4 0.51359517447216
PC5 0.540318478249624
Binary file modified sle-wb/plots/PCA/HGU133PLUS2_SCANfast_PC1-5_pairs_no.transform.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions sle-wb/plots/PCA/HGU133PLUS2_SCANfast_PC1-5_pairs_no.transform.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Principal Component cumulative variance explained
PC1 0.563109512056118
PC2 0.656474974893193
PC3 0.712575808570986
PC4 0.753517289646143
PC5 0.782092109886793
PC1 0.563497771389573
PC2 0.656798461140344
PC3 0.712814404758625
PC4 0.753738041186644
PC5 0.782282742965279
Binary file modified sle-wb/plots/PCA/HGU133PLUS2_SCANfast_PC1-5_pairs_zto.after.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions sle-wb/plots/PCA/HGU133PLUS2_SCANfast_PC1-5_pairs_zto.after.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Principal Component cumulative variance explained
PC1 0.534516911086129
PC2 0.647483735366384
PC3 0.684202498446857
PC4 0.717176853931279
PC5 0.734709417928961
PC1 0.534857479663515
PC2 0.647655030805677
PC3 0.684360488879948
PC4 0.717332073871017
PC5 0.734873202789474
Binary file modified sle-wb/plots/PCA/HGU133PLUS2_SCANfast_PC1-5_pairs_zto.before.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading