Improves search to handle smaller search terms.

[medha-14]

Description

Fixes #4734

Type of change

Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.

• New feature (non-breaking change which adds functionality)
• Optimization (back-end change that speeds up the code)
• Bug fix (non-breaking change which fixes an issue)

Key checklist:

• No style issues: $ pre-commit run (or $ nox -s pre-commit) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)
• All tests pass: $ python -m pytest (or $ nox -s tests)
• The documentation builds: $ python -m pytest --doctest-plus src (or $ nox -s doctests)

You can run integration tests, unit tests, and doctests together at once, using $ nox -s quick.

Further checks:

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.24%. Comparing base (a7253b8) to head (977f962).
Report is 24 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #4735      +/-   ##
===========================================
+ Coverage    99.22%   99.24%   +0.01%     
===========================================
  Files          303      303              
  Lines        23070    23262     +192     
===========================================
+ Hits         22891    23086     +195     
+ Misses         179      176       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[medha-14]

Could I get a review on this one?

[kratman]

@medha-14 Sorry, there is a bit of a backlog due to the upcoming release and everyone coming back from vacation. Don't worry, we will review this shortly

[medha-14]

Thank you for the update! I just thought you missed this one.

[agriyakhetarpal]

Thanks! I think min_similarity could be slightly higher here because the threshold here is too low and can lead to false positives in the search. But, at the same time, being able to resolve potential typos in the search query would require a lower threshold. This makes me feel that we could expose it with a sensible default, but I don't yet know what a sensible default would be. It should be higher than the current 40%, though. In-line comment about this below:

[agriyakhetarpal]

min_similarity is also defined above, so it gets defined twice. How about making it an argument for _find_matches()?

Follow-up question: do you think it would make sense to expose it publicly for users through search() as well?

[medha-14]

Thanks! I think min_similarity could be slightly higher here because the threshold here is too low and can lead to false positives in the search. But, at the same time, being able to resolve potential typos in the search query would require a lower threshold. This makes me feel that we could expose it with a sensible default, but I don't yet know what a sensible default would be. It should be higher than the current 40%, though. In-line comment about this below:

Sorry for the delayed response. I think it's important to clarify that min_similarity is only relevant for substring matches when the search_key is found within the known keys. For cases where there are typos or no substring matches, difflib.get_close_matches() handles those independently, using its own cutoff threshold. Increasing the min_similarity too high would make it difficult to get even the relevant matches.
Eg: If we even set the threshold to 0.5 and a user searches for conc it will not be matched with concentration because it will have similarity ratio of 0.47(approximately) which will not qualify the threshold.