New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

elo for repair #3

Open

andre15silva opened this issue Sep 16, 2024 · 1 comment

Member

andre15silva commented Sep 16, 2024

I believe so (not an export on elo system tho)

We would need:

A couple of magic numbers (starting elo, and scaling parameters for the influence of each match and computing the expected outcome)
Define a winning criteria (e.g., given 10 non-deterministic answers, which generates more correct patches)
Simulate matches until convergence

When a new LLM is added, we run the simulation again (on top of the existing results potentially).

Member Author

andre15silva commented Sep 16, 2024

https://lmsys.org/blog/2023-05-03-arena/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment