Add a 5k token buffer before the end of the context window

[mrubens]

Context

Follow-up on #1275 to give more of a buffer, just because it's currently so painful to recover from the case where you go beyond the context window and tiktoken is not an exact estimate.

Implementation

This truncates when you're within 5,000 tokens of the end of the window.

How to Test

Get a conversation near the end and confirm you never make it to within 5,000 tokens of the end of the window.

---

Important

Introduces a 5,000 token buffer in index.ts to prevent exceeding the context window, with updated tests in sliding-window.test.ts.

• Behavior:
  • Adds TOKEN_BUFFER constant in index.ts to introduce a 5,000 token buffer before the context window limit.
  • Modifies truncateConversationIfNeeded() in index.ts to truncate messages if within TOKEN_BUFFER of the token limit.
• Tests:
  • Updates sliding-window.test.ts to reflect new buffer logic, ensuring truncation occurs when within 5,000 tokens of the limit.
  • Adds test case in sliding-window.test.ts to verify truncation when tokens are within TOKEN_BUFFER of the threshold.

^{This description was created by for 764d963. It will automatically update as commits are pushed.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 764d963

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR