Retrieval evaluator with 0 scores

[sroy-forr]

I started this on a different discussion but now I know the source of why I am getting 0 scores for retrieval tasks.

import pytrec_eval

map_string="map_cut.1,3,5,10,20,100,1000"
ndcg_string="ndcg_cut.1,3,5,10,20,100,1000"
recall_string="recall.1,3,5,10,20,100,1000"
precision_string="P.1,3,5,10,20,100,1000"

qrels = {'0': {'0': 1}, '1': {'1': 1}, '2': {'2': 1}}
results = {'0': {'1': 0.730070948600769, '2': 0.730070948600769}, 
           '1': {'0': 0.6978487372398376, '2': 0.7519575357437134}, 
           '2': {'0': 0.7775193452835083, '1': 0.8888266086578369}}

evaluator = pytrec_eval.RelevanceEvaluator(
    qrels, {map_string, ndcg_string, recall_string, precision_string}
)
scores = evaluator.evaluate(results)
scores

As far as I understand it could be an issue with pytrec_eval as it seems to expect the same id to be present as well in the results dictionary (we are popping that out from results).
https://github.com/embeddings-benchmark/mteb/blob/main/mteb/evaluation/evaluators/RetrievalEvaluator.py#L151-L169

If you run the above code, you can see why I am getting 0 scores.

Any help is appreciated! Thank you!

[sroy-forr]

@KennethEnevoldsen Any suggestions on this?

Instead of the above results in the question, if I do the one that is below, I get proper values. Notice I just added the same ids with 1 score as well. Maybe the dimension matters to pytrec_eval. But why this isn't happening for others is what bothers me.

results = {'0': {'0': 1, '1': 0.730070948600769, '2': 0.730070948600769}, 
           '1': {'0': 0.6978487372398376, '1': 1, '2': 0.7519575357437134}, 
           '2': {'0': 0.7775193452835083, '1': 0.8888266086578369, '2': 1}}

[KennethEnevoldsen]

@sroy-forr this seems like an issue outside the scope of this repo I would suggest creating an issue on pytrec_eval. That being said it seems like you are confusing the corpus ids with the query ids. E.g. in the following I changed out the subset of "a", "b" and "c" to make it more clear:

import pytrec_eval
map_string="map_cut.1,3,5,10,20,100,1000"
ndcg_string="ndcg_cut.1,3,5,10,20,100,1000"
recall_string="recall.1,3,5,10,20,100,1000"
precision_string="P.1,3,5,10,20,100,1000"
qrels = {'a': {'0': 1}, 'b': {'1': 1}, 'c': {'2': 1}}
results = {'a': {'1': 0.730070948600769, '2': 0.730070948600769, "0": 0}, 
           'b': {'0': 0.6978487372398376, '2': 0.7519575357437134, "1": 0}, 
           'c': {'0': 0.7775193452835083, '1': 0.8888266086578369, "2": 0}}

evaluator = pytrec_eval.RelevanceEvaluator(
    qrels, {map_string, ndcg_string, recall_string, precision_string}
)

scores = evaluator.evaluate(results)
scores

Naturally since your corpus contains 3 documents you should have a relevance score for each of them or at least have the correct document, which was missing in all cases it seems. In fact you will see that this also works:

import pytrec_eval
map_string="map_cut.1,3,5,10,20,100,1000"
ndcg_string="ndcg_cut.1,3,5,10,20,100,1000"
recall_string="recall.1,3,5,10,20,100,1000"
precision_string="P.1,3,5,10,20,100,1000"
qrels = {'a': {'0': 1}, 'b': {'1': 1}, 'c': {'2': 1}}
results = {'a': {"0": 1}, # perfect model
           'b': { "1": 1}, 
           'c': { "2": 1}}

evaluator = pytrec_eval.RelevanceEvaluator(
    qrels, {map_string, ndcg_string, recall_string, precision_string}
)

scores = evaluator.evaluate(results)
scores

[sroy-forr]

Thank you for answering! I made the following code changes to make them work.

Removed this line: https://github.com/embeddings-benchmark/mteb/blob/main/mteb/evaluation/evaluators/RetrievalEvaluator.py#L157

Set this: ignore_identical_ids=False

I am not sure about any confusion between corpus and query id as I just followed the documentation. But surely I am confused as to why I had to make code changes while it worked for everyone else :-D.

I am marking your answer. Appreciate it!