feature: Addition of memory snippets #82
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nisearch and modifying search to use multiple differently prioritised segments
BBC-Esq
reviewed
Aug 5, 2024
BBC-Esq
reviewed
Aug 5, 2024
|
Hope I did this right... |
…f fixes for search results
|
Was there something to review or you mentioned that you fixed the issue? |
|
I have yet to try the improved version, which I will review once you two are done with it |
BBC-Esq
approved these changes
Aug 7, 2024
As discussed in the chat on Discord, the requested settings should now be UI options. Henky is giving it a quick go I believe, and the library used can be found here: https://github.com/lucaong/minisearch And the minified build in particular here: https://www.jsdelivr.com/package/npm/minisearch (https://cdn.jsdelivr.net/npm/[email protected]/dist/umd/index.min.js) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A variant of the ideas of World Info and Vector based Long Term Memory. While within Lite we cannot expect vector storage to exist - at least not in its traditional form of embeddings any time soon, it is possible to implement the searching algorithms they use to extract information from the entire game text which are relevant. These are based on similarity, and after testing a few I settled on Tf-Idf - which is explained more in the links I provided in the comments within the new code.
The general basis looks for frequency of distinctive words, the snippet length and a couple of factors like that - to try and provide a better search while still remaining performant. While this is still a fairly rudimentary implementation / integration within Lite, it runs entirely browser side and seems to do alright with finding relevant matches.
With the implementation, there is the addition of a new menu under context which allows enabling the new option, modifying the minimum similarity required to find matches and how many matches to include (similar to Top N). The matches are included with the world info as an authors note formatted block with the different snippets.
While I have tried to generally match formatting / style with the existing code, as this is my first PR for Lite apologies in advance if there's anything out of place!