Metrics covered in the book

[santiviquez]

Here, we keep track of all the metrics we plan to cover in the book and their status. Each one should reference an issue.

Note: Some of these issues are already 'solved' since many metric pages have already been written. But I still need to convert them to LaTeX format, so they're still open here. If you see an issue assigned to me, I’ve probably already written the content for that metric.

Regression

• MAE — Mean Absolute Percentage Error #1
• MSE — mean squared error #2
• MSLE — Mean Squared Log Error #3
• RMSE — Root Mean Squared Error #4
• RMSLE — Root Mean Squared Log Error #5
• RMSLE — Root Mean Squared Log Error #5
• MAPE — Mean Absolute Percentage Error #6
• sMAPE — Symmetric Mean Absolute Percentage Error #7
• wMAPE — Weighted Mean Absolute Percentage Error #8
• MASE — Mean Absolute Scaled Error #9
• MDA — Mean Directional Accuracy #11
• MPD — Mean Poisson Deviance #12
• MGD — Mean Gamma Deviance #13
• R-squared #14
• D2 — D2 Absolute Score #15
• Explained Variance Score #16
• Continuous Ranked Probability Scores (CRPS)  #111

Classification

• Confusion Matrix #17
• FPR — False Positive Rate #18
• FNR — False Negative Rate #19
• FNR — False Negative Rate #19
• TPR — True Positive Rate (Recall/Sensitivity) #20
• TNR — True Negative Rate (Specificity) #21
• Accuracy #22
• Balanced Accuracy #23
• Precision #24
• F1-score #25
• F-beta #26
• ROC AUC — Area Under the Receiver Operating Characteristic Curve #27
• PR AUC — Area Under the Precision-Recall Curve #28
• Brier Score Loss #29
• Log Loss #30
• Jaccard Score #31
• D2 Log Loss Score #32
• P4-metric #33
• Cohen's Kappa #34
• Phi Coefficient #35
• MCC — Matthew's Correlation Coefficient #36
• Expected Cost (EC) #94

Clustering

• Mutual Info Score #37
• Adjusted Mutual Info Score #38
• Normalized Mutual Info Score #39
• Rand Score #40
• Adjusted Rand Score #41
• Calinski Harabasz Score #42
• Contingency Matrix #43
• Pair Confusion Matrix #44
• Completeness Score #45
• Davis Bouldin Score #46
• Fowlkes Mallows Score #47
• Homogeneity Score #48
• V Measure #49
• Homogeneity Completeness V Measure #50
• Silhouette Score #51
• Consensus Score #52
• Dunn Index #106

Ranking

• @K metrics #181
• MRR — Mean Reciprocal Rank #53
• MAP — Mean Average Precision #61
• nDCG@K — Normalized Discounted Cumulative Gain #55
• Hit Rate
• Intra-list Diversity
• Coverage
• Behavioral Metrics (Novelty, Serendipity, Diversity)

Computer Vision

This needs further discussion, as depending on the use case, some computer vision metrics overlap with other categories. For example, image classification tasks can be evaluated using standard classification metrics. The goal of this section is to focus on evaluation metrics that are specifically related to computer vision.

• Pixel Accuracy #62
• Mean Pixel Accuracy #63
• Boundary IoU #64
• PCK — Percentage of Correct Keypoints #65
• OKS — Object Keypoint Similarity #66
• IS — Inception Score #67
• FID — Frechet Inception Distance #68
• PSNR - Peak Signal-to-Noise Ratio #91

NLP

Similar to computer vision, this needs further discussion, as depending on the use case, some NLP metrics overlap with other categories. For example, text classification tasks can be evaluated using standard classification metrics. The goal of this section is to focus on evaluation metrics that are specifically related to NLP.

• BLEU — Bilingual Evaluation Understudy #70
• METEOR — Metric for Evaluation of Translation with Explicit ORdering #71
• ROUGE — Recall-Oriented Understudy for Gisting Evaluation #72
• TER — Translation Edit Rate #73
• EM — Exact Match #74

GenAI

This needs further discussion. Feel free to suggest any metric by opening an issue.

• Perplexity #175

Probabilistic

• CBPE — Confidence-based Performance Estimation #75
• DLE — Direct Loss Estimation #76
• PAPE — Probabilistic Adaptive Performance Estimation #77
• IW — Importance Weighting #78

Bias & Fairness

• Demographic Parity #83
• Equality of Opportunity #84
• Equalized Odds #85
• Predictive Parity #86
• Parity Measures #87
• Calibration within Groups #88
• Treatment Equality #89
• Disparate Impact #90

Business

This list could be infinite. The idea is to cover the top 3-5 metrics for relevant industries that have adopted ML and use non-traditional metrics to evaluate their models. If you're an industry expert working on ML, we would value your input.

• MTBF — Mean Time Between Failures #79
• MTTF — Mean Time To Failure #80
• MTTR — Mean Time To Repair #81
• Sharpe Ratio #82
• Conversion Rate #95
• Bounce Rate #96

Any other suggestions? Leave a comment or open an issue.