Add MTEB output for Google's text-embedding-005

[chenblair]

Output is generated by in-house Google evaluation script forked off of original MTEB evaluation code

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the results files checker make pre-push.

[Muennighoff]

Amazing! @KennethEnevoldsen is external the right folder? When naming the folder no_revision_available the tests failed for @chenblair. I had assumed that 1 would be the wrong folder to put it in since that one is only for results run with https://github.com/embeddings-benchmark/mteb/blob/25f4f618f1694d1155919c9771c551fa70b5049b/mteb/models/google_models.py#L157 while these results are probably run with the internal Google implementation. Or is 1 fine?

[KennethEnevoldsen]

Amazing! @KennethEnevoldsen is external the right folder?

External should be fine, unless it is in fact run with our implementation then it is in fact 1. If not I would recommend that they make a PR of their implementation (allows users to know how they should use the API).

Output is generated by in-house Google evaluation script forked off of original MTEB evaluation code

Explains some of the point. Would rerun it using the latest version of the public code-base.

It seems like you are already using the implemented wrapper (from the model_meta.json):

https://github.com/embeddings-benchmark/mteb/blob/main/mteb/models/google_models.py

So I would assume that it should work out of the box.

[chenblair]

Thanks for the detailed comments and discussion! I will look into making a PR of our current implementation, the metrics as is should be ready to merge.

[KennethEnevoldsen]

Fixes look good, I have this in

[chenblair]

@Muennighoff

I'm not currently seeing the text-embedding-005 metrics on the main MTEB leaderboard (https://huggingface.co/spaces/mteb/leaderboard). Is there anything else I need to do to have this model be populated on the leaderboard?

Thanks!

[Muennighoff]

I think the paths.json file hasn't been updated? You need to run this function from the source of the repo

results/results.py

Line 298 in c33da34

def get_paths():

I think? cc @KennethEnevoldsen

The new leaderboard will no longer require that and I think it already shows up there (http://mteb-leaderboard-2-demo.hf.space/?benchmark_name=MTEB%28eng%2C+classic%29)

[Samoed]

I think the paths.json file hasn't been updated? You need to run this function from the source of the repo.

Also to be added to old leaderboard you need to update model_meta.yaml and run refresh.py