docs: Add Semantic Caching Tutorial #118
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
oandreeva-nv
requested review from
nnshah1,
rmccorm4,
GuanLuo,
kthui and
krishung5
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
| supported feature in Triton Inference Server. | ||
|
|
||
| We value your input! If you're interested in seeing semantic caching as a | ||
| supported feature in future releases, we encourage you to [FILL IN] |
There was a problem hiding this comment.
FILL IN
reminder for self
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
Comment on lines
+313
to
+315
| Clearly, the latter 2 requests are semantically similar to the first one, which | ||
| resulted in a cache hit scenario, which reduced the latency of our model from | ||
| approx 1.1s to the average of 0.048s per request. |
There was a problem hiding this comment.
Did you have any rough ideas of cache miss cost? ex: 1 request without semantic caching vs 1 request with semantic caching? Just curious on the rough magnitude of the overhead.
There was a problem hiding this comment.
I can probably do some estimations for that.
There was a problem hiding this comment.
I'm leaving this unresolved to remind myself to do this study as a follow-up
rmccorm4
reviewed
Oct 23, 2024
rmccorm4
reviewed
Oct 23, 2024
Co-authored-by: Ryan McCormick <[email protected]>
krishung5
reviewed
Oct 23, 2024
|
Nice tutorial! Really fun to try it out 🚀 |
GuanLuo
previously approved these changes
Oct 23, 2024
krishung5
previously approved these changes
Oct 23, 2024
rmccorm4
reviewed
Oct 25, 2024
rmccorm4
reviewed
Oct 25, 2024
rmccorm4
reviewed
Oct 25, 2024
rmccorm4
reviewed
Oct 25, 2024
rmccorm4
reviewed
Oct 25, 2024
rmccorm4
previously approved these changes
Oct 25, 2024
rmccorm4
changed the title
Co-authored-by: Ryan McCormick <[email protected]>
rmccorm4
approved these changes
Oct 25, 2024
krishung5
approved these changes
Oct 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds reference implementation of local Semantic caching mechanism.
Note:
Adding a CPU based index, since for current tutorial this seems enough. GPU-based index makes more sense when we have large amount of vectors to process.
Opens:
I've added a section called
Interested in This Feature?, which is not finished at the moment.My suggestion for community engagement is to create a GitHub issue and encourage readers and users to vote on it, in case it is interested.
If there're no objections, I'll proceed with this idea.
[Edit 1] discussion opened -> triton-inference-server/server#7742