Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Story]: Display Prompt cache statistics #940

Open
kantord opened this issue Feb 5, 2025 · 0 comments
Open

[Story]: Display Prompt cache statistics #940

kantord opened this issue Feb 5, 2025 · 0 comments

Comments

@kantord
Copy link
Member

kantord commented Feb 5, 2025

Description

Depends on #939 and #892, as until those are fixed the utility of this feature is minimal or none.

Anthropic and OpenAI provide prompt caching features, and their APIs return statistical data about the savings using prompt caching.

As a cost-conscious or environmentally conscious user, I would like to know that I am properly using prompt caching during my daily work, and how much efficiency is gained.

I would also like to be able to see if one of the AI assistants or tools that I am using prevents this feature.

I would like to be able to access this conveniently on the CodeGate dashboard.

Additional Context

We may have to normalize this into a common struct in order to make it usable across providers

For a good starting point on prompt caching in general, you can check out this internal document: https://docs.google.com/document/d/1gEgkIyT1CRcTMAwYiU6ldeeosKKbWMX5HM8AdQx3xV0/edit?tab=t.0

@kantord kantord changed the title [Story]: Prompt cache statistics [Story]: Display Prompt cache statistics Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants