Which tokenization technique is employed by BM25? #1877

Closed Unanswered

lxx1220 asked this question in Q&A

lxx1220
Apr 30, 2024

Replies: 1 comment

lintool
Apr 30, 2024
Maintainer

Depends on language. For English: DefaultEnglishAnalyzer https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/index/IndexCollection.java#L62

For other languages: https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/index/IndexCollection.java#L213

0 replies

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment