[Bug]: Litellm proxy doesn't support caching unless used with litellm_proxy prefix

[enyst]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Describe the bug and reproduction steps

Reported on slack : the .jsonl after a swe-bench run reports 0.

Anthropic claude-sonnet-3-5-20241022 via litellm_proxy.

OpenHands Installation

Other

OpenHands Version

No response

Operating System

None

Logs, Errors, Screenshots, and Additional Context

No response

[mamoodi]

Does that mean prompt caching is broken right now? Or is this only relevant to evaluation?

[enyst]

Relevant: conclusion on slack

It seems that when using a litellm proxy, the user needs to call the model with the prefix litellm_proxy, otherwise it won't apply caching. I find it strange (we did set the cache markers, it must be that the proxy is removing them for some calls?), but it doesn't affect everyone, only proxy users.