Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mean reciprocal rank (MRR) of leaderboard scores #3452

Merged
merged 2 commits into from
Dec 8, 2023
Merged

Mean reciprocal rank (MRR) of leaderboard scores #3452

merged 2 commits into from
Dec 8, 2023

Conversation

Didayolo
Copy link
Collaborator

@Didayolo Didayolo commented Dec 8, 2023

Original PR

Purpose of the changes contained in this PR.

For a new competition that we will launch soon on CodaLab, we would like to use the mean reciprocal rank (MRR) as a summary metrics for the overall evaluation of a group of scores in the leaderboard. However, at the moment, Codalab only allows using the (weighted) average operation over the rank of scores.
This is mentioned in some comments on #2736 and #3449.

Issues this PR resolves

This PR includes the operation to calculate the MRR over leaderboard scores.

A checklist for hand testing

  • create a competition bundle with a column of the leaderboard performing the MRR of some metrics. You can use the modified IRIS competition bundle.
  • upload a few submissions to the competition.
  • check that the MRR column provides the correct calculation and ranking, similar as in the screenshot below.
    Screenshot 2023-11-30 at 10 25 25

Any relevant files for testing

I have created a modified IRIS competition bundle that has, as leaderboard columns, both the MRR and the Avg of the original metrics (Prediction Score and Duration).

Checklist

  • Code review by me and @JesusPinedaC
  • Hand tested by me
  • I'm proud of my work (code changes are very minor though!)
  • Code review by reviewer
  • Hand tested by reviewer
  • CircleCi tests are passing
  • Ready to merge

@codecov-commenter
Copy link

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (927279f) 60.45% compared to head (3c28c2f) 60.38%.

Files Patch % Lines
codalab/apps/web/models.py 23.52% 13 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3452      +/-   ##
===========================================
- Coverage    60.45%   60.38%   -0.07%     
===========================================
  Files          173      173              
  Lines        11396    11411      +15     
===========================================
+ Hits          6889     6891       +2     
- Misses        4507     4520      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Didayolo Didayolo merged commit 6901945 into develop Dec 8, 2023
2 checks passed
@Didayolo Didayolo deleted the mrr branch December 8, 2023 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants