Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Add Rank based combination technique for Hybrid Search #659

Open
vamshin opened this issue Apr 1, 2024 · 0 comments
Open

[FEATURE] Add Rank based combination technique for Hybrid Search #659

vamshin opened this issue Apr 1, 2024 · 0 comments
Assignees

Comments

@vamshin
Copy link
Member

vamshin commented Apr 1, 2024

Is your feature request related to a problem?

Hybrid search's current normalization techniques considers scores to bring different sub queries to same scale. Using this approach, the top-ranked result will invariably receive a score of 1, while the lowest-ranked result will consistently be assigned a score of 0, regardless of their proximity or distance from the search query. Integrating the normalized scores with additional sub-queries can potentially introduce bias, as the score distribution may influence the overall results for certain data sets.

What solution would you like?

As an alternative normalization and combination technique, Reciprocal Rank Fusion (RRF) for Hybrid Search utilizes the positions or ranks of the documents rather than relying solely on their scores. This approach aims to mitigate potential biases that may arise from combining scaled scores with other sub-queries.

@vamshin vamshin moved this from Backlog to Backlog (Hot) in Vector Search RoadMap Apr 1, 2024
@vamshin vamshin removed the untriaged label Apr 1, 2024
@vamshin vamshin changed the title [FEATURE] Add Reciprocal Rank Fusion(RRF) for Hybrid Search as a new normalization/combination technique [FEATURE] Add Rank based combination technique for Hybrid Search Apr 1, 2024
@vamshin vamshin moved this from Backlog (Hot) to Now(This Quarter) in Vector Search RoadMap Apr 1, 2024
@navneet1v navneet1v moved this from Now(This Quarter) to Backlog (Hot) in Vector Search RoadMap Jun 14, 2024
@vamshin vamshin moved this to 2.17.0 in Vector Search RoadMap Jul 2, 2024
@naveentatikonda naveentatikonda moved this from 2.17.0 to Now(This Quarter) in Vector Search RoadMap Aug 29, 2024
@vamshin vamshin added the Roadmap:Vector Database/GenAI Project-wide roadmap label label Aug 29, 2024
@martin-gaievski martin-gaievski moved this from 2.18.0 to 2.19.0 in Vector Search RoadMap Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: 2.19.0
Development

No branches or pull requests

3 participants