Add personal leaderboard

[Muennighoff]

I think we should add a personal leaderboard like in this arena: https://artificialanalysis.ai/text-to-image/arena (also see screenshot) as:

• allows users to know even better what matches their needs, not what matches the average person's need
• incentivizes more voting as we'd need a threshold of say 30 votes for the personal LB
• makes for a better preference dataset as we can group them by user - e.g. these preferences are from user A; these from user B etc (can be anonymized); also users could collect a dataset of their own preferences which could be useful for them

Does someone have thoughts on this / bandwidth to work on this? 🙌 cc @isaac-chung @orionw & anyone else interested :)

[orionw]

This is a neat idea! I'm curious how you envision it working with the huggingface space though? E.g. how would we match a user to their data? AFAIK I don't think gradio keeps cookies or has logins (though would be really cool if they did).

[Muennighoff]

Yeah I think we need to check if we can do it via IP else have users provide their username (disadvantage being that you could submit for other people's usernames but unlikely many people would do that I think). Then we just log the username (maybe hashed) as part of the vote data.

[malteos]

Gradio supports OAuth: https://www.gradio.app/guides/sharing-your-app#o-auth-login-via-hugging-face

So you could let people login via their HF account.

[Muennighoff]

Gradio supports OAuth: https://www.gradio.app/guides/sharing-your-app#o-auth-login-via-hugging-face

So you could let people login via their HF account.

Oh this is amazing! Would you be interested in adding this feature?

[Muennighoff]

We're lacking votes a bit in the arena with only one vote every 1-2 hours atm - Does someone have bandwidth to look into this issue, maybe @orionw or @isaac-chung ? 🙌 Imo it's the best idea for increasing our votes & not having to take the arena down again because the cost (~760 USD/day) vs traction is not good enough.. am in discussions with sponsors..

[KennethEnevoldsen]

I sadly do not have the time atm. (PhD dissertation, maintaining mteb, mmteb). I will have more time once mmteb is finalized. However, do agree that is it probably the way to go.

How was the traction in the first few days, is it worth doing another round of SoMe? We might be able to push to 1-2 up to 3-5. What are the sponsors setting as a reasonable target?

1-2 gives us 1.5 * 24h * 30days = ~1000/month

[isaac-chung]

+1 on another social media push. Did we end up getting retweets that we wanted btw (e.g. 🤗) ? I also have the same question as Kenneth re: targets.

Costs: Could we get away with only keeping one corpus (e.g. wiki) online, or maybe we're already doing that? For num replicas, maybe we go to minimum=0? The first request will take longer but at least that should save some $$.

[aamir-s18]

How about moving the inference to CPU to reduce the cost for most models with e.g. int8 it should be feasible to have an acceptable response time?