whoosh library is outdated

[jdangerx]

Overview

The whoosh library we've been using for our keyword search is not maintained (thanks to @krivard for pointing this out!). It would be good to switch to something that is.

But, I also want to avoid deploying a whole new service to search for... 200 documents.

Looks like our main options are:

• whoosh-reloaded - a maintained version of whoosh, though it doesn't seem that maintained. Latest update was about a year ago, and despite releasing version 3.0.0 the docs all still say 2.7.4.
• xapian-bindings-binary - python bindings for the xapian search engine. old-school, 'battle-tested', but also kind of clunky
• tantivy-py - python bindings for tantivy. new-school, basically a rust clone of lucene. An API that immediately made sense to me vs. the Xapian one.
• dark horse: use sentence-transformers for semantic search!

At this point I think the last two are the most appealing.

Success Criteria

• we don't depend on whoosh
• search still returns reasonable results

Next steps

Preview Give feedback

1. write an integration test to make sure that search endpoint is giving back reasonable HTML
2. prototype using sentence-transformers and tantivy

[zaneselvans]

Ah, the cycle of life (OSS). 👶🏼👧🏼👩🏼👵🏼🧟‍♀️