-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feat
] Integrate NanoBeIR datasets
#2966
base: master
Are you sure you want to change the base?
[feat
] Integrate NanoBeIR datasets
#2966
Conversation
Although the |
I'm experimenting with having all outputs in the final dict, rather than a nested dict. This way, people can use any value from the evaluator to guide their e.g. early stopping. It should also match the I hope it's okay if I push into this PR! |
As discussed in #2848 (comment), This PR adds a new Evaluator based on the NanoBEIR collection of datasets.
It creates one
InformationRetrievalEvaluator
for each dataset, and aggregates the results accordingly.Example:
(Note that this depends on #2951)