MTEB Leaderboard

[do-me]

The Massive Text Embedding Benchmark (MTEB) Leaderboard shows that there might be some interesting models to replace all-MiniLM-L6-v2 as default model.

E.g. bge-small-en with only 134Mb (unquantized!) looks promising. It has only 384 dimensions which is important to lower memory usage.

But there are other candidates too with even smaller sizes:

E.g. gte-small with only 67Mb (unquantized).

[do-me]

I just checked the models in descending order for three things:

• multilingual training data
• low dimensionality (768 or below)
• low size (1Gb or below)

and the winner seems to be multilingual-e5-small.

Even though the multilingual fine tuning data is very little in comparison to EN (and ZH), it seems to outperform our current multilingual model distiluse-base-multilingual-cased-v2. Apart from that, it's smaller and has less dimensions. Also, the issue with dense layers in SentenceTransformers (Python, 512 dimensions vs. without dense layer 768 dimensions) shouldn't apply (still need to check though).

[do-me]

Referencing #36 for completeness; default model is now gte-small.

[do-me]

SemanticFinder dropdown model selector

@MentalGear as you were asking about the model dropdown: well it grew organically. It started with a hand full of models, then I automated the model mining process from HF with likes and downloads but quickly noticed, that it was annoying me that new but high-scoring models were at the bottom, so I switched the order a little with my personal most-used models on top. So pretty subjective to my personal preferences.

I was thinking about providing only a text input and leaving the model choice entirely to the user but it might be frustrating when there are typos or similar. Also, atm I test all models I integrate so at least I can make sure all models do work too.

I might add some kind of model selection help in the future but I don't want to reproduce MTEB. Instead, maybe a short (and subjective) guideline would be more appropriate... Any ideas welcome!

[MentalGear]

Results by Trial & Error :)

Definitely a better UX to have guidance than some other freeform transformer demos.

Did you find that the models you preferred for the semantic-finder generally aligned to the MTEB?

[do-me]

I would even put it differently: it's hard to find any well-ranking MTEB model that performs significantly worse than others.

If they do, first one should head over to the repo and check whether they require "query: " / "passage: " prefixes or similar and if the onnx conversion went fine.

They do differ in nuances though, which is of course expected. The only poor results I found so far were with multilingual models where the same word or phrase in different languages didn't quite align as expected.

Apart from that, I'm most excited about really small models showing impressive results in comparison to 10x larger models! E.g. gte-small and bge-small-en are fantastic.

[MentalGear]

Yeah, small highly reliant models would be great for offline-first.

In the semantic-finder demo, when a demo text is loaded in, is the vectorization done at that moment, or is only a vector index loaded in?

[do-me]

It's done on the fly in the browser but there is the possibility to export and import the index afterwards like I did for a few examples. When you import an index only the search query needs inferencing. This makes subsequent queries fairly fast.

[MentalGear]

Neat to see that vectorization works so fast now. A few months ago I also started developing a browser extension with Transformers js to semantic sort tabs, but I remember that vectorization of tab titles with urls took quite a bit longer.

What do you think of langchain.js ?