Cache usage information when streaming from Anthropic

[CarnageXD]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

const model = new ChatAnthropic({
      apiKey: process.env.ANTHROPIC_API_KEY,
      temperature: 0.2,
      verbose: true,
      streaming: true,
      streamUsage: true,
      model: process.env.ANTHROPIC_MODEL,
      callbacks: [
        {
          handleLLMNewToken: (token, idx, runId, parentRunId, tags, fields) => {
            console.log('token', token);
            console.log('fields', fields);
            res.write(token);
          },
        },
      ],
      clientOptions: {
        defaultHeaders: {
          'anthropic-beta': 'prompt-caching-2024-07-31',
        },
      },
    });

Description

When I use Anthropic without streaming, I can get information about the cache usage from the model response from the "response_metadata" and the "usage" field inside it. Like this:

"response_metadata": {
    "id": "msg_01GFAf23N8GToFD7Cs1J1Wk1",
    "model": "claude-3-5-sonnet-20240620",
    "stop_reason": "end_turn",
    "stop_sequence": null,
    "usage": {
        "input_tokens": 429,
        "cache_creation_input_tokens": 0,
        "cache_read_input_tokens": 0,
        "output_tokens": 174
    "type": "message",
    "role": "assistant"
}

But when I use streaming, in this case, the "response_metadata" field is empty and the only way to check tokens usage is to look at the "usage_metadata" field, but it doesn't have information about the cache:

AIMessageChunk {
  "id": "msg_01F79VPWQFNPzF8meQVPC6o8",
  "content": "some content",
  "additional_kwargs": {
    "id": "msg_01F79VPWQFNPzF8meQVPC6o8",
    "type": "message",
    "role": "assistant",
    "model": "claude-3-5-sonnet-20240620",
    "stop_reason": "end_turn",
    "stop_sequence": null
  },
  "response_metadata": {},
  "tool_calls": [],
  "tool_call_chunks": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 434,
    "output_tokens": 119,
    "total_tokens": 553
  }
}

Also, when I log every single chunk of data, the result is the same – "response_metadata" is empty, and only the "usage_metadata" is the option to see, but it doesn't have cache usage info

Is this something missing from LangChain's implementation, or am I looking in the wrong place?

System Info

"@langchain/anthropic": "^0.3.5",
"@langchain/core": "^0.3.13",
"langchain": "^0.3.3"

node v20.11.0

[dosubot[bot]]

Hey @CarnageXD! I'm here to help you with your issue. I'm a bot that can assist you with bug fixes, answer questions, and guide you on becoming a contributor. Let's tackle this together!

I found a relevant discussion that might help you with your issue. In LangChain, when streaming from Anthropic models, the response_metadata field can indeed be empty, and the usage_metadata is used to track token usage. However, the usage_metadata does not include cache usage information by default.

There are configuration options in LangChain that affect the presence of cache usage information in the response_metadata when streaming. Specifically, you can use the stream_options parameter to control this behavior. Setting stream_options: { include_usage: true } includes usage metadata in the response, but it seems that cache usage information might not be included in the streaming response by default ^[1].

If you need cache usage information during streaming, you might need to check if there are any updates or additional configurations in the LangChain library that could enable this feature. As of the information available, it seems that the current implementation might not support cache usage details in the usage_metadata during streaming.

To continue talking to Dosu, mention @dosu.

[CarnageXD]

@dosu Anthropic does not have such an option for its model, it has "streamUsage" which does the same thing as far as I understand

[dosubot[bot]]

The "streamUsage" option in Anthropic's API does provide cache usage information during streaming. When you set streamUsage: true in the call options, you can access the usage metadata, which includes details like input_tokens, output_tokens, and total_tokens. This confirms that the "streamUsage" option provides cache usage information during streaming ^[1][2].

To continue talking to Dosu, mention @dosu.

[nikeshparajuli]

Experiencing the same. Anthropic says the following:

Monitor cache performance using these API response fields, within usage in the response (or message_start event if streaming)

So it looks like the first response in the stream should have these parameters but I can't see them when I log the responses. It may not have been implemented yet in LangChain.

[CarnageXD]

@nikeshparajuli Yeah, I switched to the Anthropic SDK (it has all data) temporarily until LangChain adds it to their package