lrgasp-challenge-2-evaluation

Web application

We have a web application that illustrates the evaluation plots and functions here.
For a live video demo of this web app, you can check here.

Requirements

Running python 3.7 and linux64 on your machine

Install

git clone https://github.com/Tidesun/LRGASP_visualization.git
cd LRGASP_visualization
python -m venv base
source base/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

Input

• Gene/isoform annotation: GTF format
  • Human and Mouse reference annoation with spike-in at Link
• Quantification result:

# Sample	# Methods	Format	Columns	Example Data Path
Single	Single	TSV	First column: ID Second column: Quanficiation result for single sample	single sample data
Multiple	Single	TSV	First column: ID Next N columns: Quantification results for multiple samples	multiple samples data
Single	Multiple	ZIP	First column: ID Second column: Quanficiation result for single sample	multiple methods with single sample
Multiple	Multiple	ZIP	First column: ID Next N columns: Quantification results for multiple samples	multiple methods with multiple samples

* TSV format is defined in (https://github.com/LRGASP/lrgasp-submissions/blob/master/docs/expression_matrix_format.md)
* Multiple methods result files should be named as the method name and the output will be named accordingly.

• Expression ground truth:
  • TSV format as defined in (https://github.com/LRGASP/lrgasp-submissions/blob/master/docs/expression_matrix_format.md).

Output

• Report webpage: HTML format
• Evaluation graphs: PNG and PDF format

Section	Single sample	Multiple sample with Ground Truth	Multiple sample without Ground Truth
Gene features	Plots: Distribution of K values Distribution of isoform lengths Distribution of numbers of exons	Plots: Distribution of K values Distribution of isoform lengths Distribution of numbers of exons	Plots: Distribution of K values Distribution of isoform lengths Distribution of numbers of exons
Estimation Error	Tables: Normalized Root Mean Square Error Median Relative Difference Mean Abundance Recovery Rate Spearman's rho Plots: Histogram of Abundance Recovery Rate Correlation of estimated abundance and ground truth Correlation Boxplot of estimated abundance and ground truth	Tables: Normalized Root Mean Square Error Median Relative Difference Mean Abundance Recovery Rate Spearman's rho Plots: Estimation Error for different conditions Histogram of Abundance Recovery Rate Correlation of estimated abundance and ground truth Correlation Boxplot of estimated abundance and ground truth
Resolution Entropy	Tables: Resolution Entropy Plots: Resolution Entropy	Tables: Resolution Entropy Plots: Resolution Entropy for different conditions	Tables: Resolution Entropy Plots: Resolution Entropy for different conditions
Consistency		Tables: Consistency Measure Plots: Consistency Measure curve	Tables: Consistency Measure Plots: Consistency Measure curve
Irreproducibility		Tables: Irreproducibility Measure Plots: coefficient of variation vs estimated abundance curve	Tables: Irreproducibility Measure Plots: coefficient of variation vs estimated abundance curve
Fold change based evaluation		Tables: Precision Recall Accuracy F1 score AUC Plots: ROC curves for performance of quantification PR curves for performance of quantification
Split Statistics	Plots: Statistics with different K values Statistics with different isoform lengths Statistics with different numbers of exons Statistics with different expression level	Plots: Statistics with different K values Statistics with different isoform lengths Statistics with different numbers of exons Statistics with different expression level	Plots: Statistics with different K values Statistics with different isoform lengths Statistics with different numbers of exons Statistics with different expression level

Usage

Organizer reporter

source base/bin/activate
python encode_quantification/main.py -a ANNOTATION -r RESULT -t TRUTH -o OUTPUT --num_method NUM_METHOD  --num_samples NUM_SAMPLES

Quantification evaluation reporter

optional arguments:
  -h, --help            show this help message and exit

required named arguments:
  -a ANNOTATION, --annotation ANNOTATION
                        The path of annotation file [GTF]
  -r RESULT, --result RESULT
                        The path of quantification result file [TSV\ZIP]
  -o OUTPUT, --output OUTPUT
                        The path of output directory
  --num_method NUM_METHOD
                        Whether multi method data given ['Single' or 'Multi']
  --num_samples NUM_SAMPLES
                        Whether multi sample data given ['Single' or 'Multi']

optional arguments:
  -t TRUTH, --truth TRUTH
                        The path of true expression file [TSV]
  --seq SEQ             Whether long read data given ['LongRead' or
                        'ShortRead'] [default:LongRead]
  --K_value_selection K_VALUE_SELECTION
                        Which K value calculation['Condition_number','K_value'
                        ,'Generalized_condition_number']
                        [default:Condition_number]

Example

Single method evaluation

Single method Single sample

source base/bin/activate
python encode_quantification/main.py \
-a example/chr1.gtf \
-r example/singlesample/methodA.tsv \
-t example/singlesample/truth.tsv \
-o example/reports \
--num_method Single \
--num_samples Single

Single method Multiple sample with ground truth

source base/bin/activate
python encode_quantification/main.py \
-a example/chr1.gtf \
-r example/multisample/methodA.tsv \
-t example/multisample/truth.tsv \
-o example/reports \
--num_method Single \
--num_samples Multi

Single method Multiple sample without ground truth

source base/bin/activate
python encode_quantification/main.py \
-a example/chr1.gtf \
-r example/multisample/methodA.tsv \
-o example/reports \
--num_method Single \
--num_samples Multi

Multiple methods evaluation

Multiple method Single sample

source base/bin/activate
python encode_quantification/main.py \
-a example/chr1.gtf \
-r example/singlesample/methods.zip \
-t example/singlesample/truth.tsv \
-o example/reports \
--num_method Multi \
--num_samples Single

Multiple method Multiple sample with ground truth

source base/bin/activate
python encode_quantification/main.py \
-a example/chr1.gtf \
-r example/multisample/methods.zip \
-t example/multisample/truth.tsv \
-o example/reports \
--num_method Multi \
--num_samples Multi

Multiple method Multiple sample without ground truth

source base/bin/activate
python encode_quantification/main.py \
-a example/chr1.gtf \
-r example/multisample/methods.zip \
-o example/reports \
--num_method Multi \
--num_samples Multi

Example report screenshot