-
Notifications
You must be signed in to change notification settings - Fork 304
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(openai): support streamed responses for completions and chat completions #4308
Conversation
Overall package sizeSelf size: 6.53 MB Dependency sizes
🤖 This report was automatically generated by heaviest-objects-in-the-universe |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #4308 +/- ##
==========================================
- Coverage 85.94% 82.20% -3.75%
==========================================
Files 117 245 +128
Lines 4384 10350 +5966
Branches 33 33
==========================================
+ Hits 3768 8508 +4740
- Misses 616 1842 +1226 ☔ View full report in Codecov by Sentry. |
BenchmarksBenchmark execution time: 2024-05-24 18:31:48 Comparing candidate commit 71bf20c in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 259 metrics, 7 unstable metrics. |
9435181
to
cc4352a
Compare
* yarn * add remaining tests * remove yarn files * add yarn lock back * format * fix lint * fix test * fix lints * Update packages/datadog-plugin-openai/test/index.spec.js Co-authored-by: Yun Kim <[email protected]> --------- Co-authored-by: Yun Kim <[email protected]>
1efbb63
to
f237358
Compare
@@ -0,0 +1,5 @@ | |||
data: {"id":"chatcmpl-9S9XTKSaDNOTtVqvF2hAbdu4UGYQa","object":"chat.completion.chunk","created":1716496879,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this literally the streaming protocol that OpenAi uses?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, for the older versions
…pletions (#4308) Co-authored-by: Jonathan Chavez <[email protected]> Co-authored-by: Yun Kim <[email protected]>
…pletions (#4308) Co-authored-by: Jonathan Chavez <[email protected]> Co-authored-by: Yun Kim <[email protected]>
…pletions (#4308) Co-authored-by: Jonathan Chavez <[email protected]> Co-authored-by: Yun Kim <[email protected]>
…pletions (#4308) Co-authored-by: Jonathan Chavez <[email protected]> Co-authored-by: Yun Kim <[email protected]>
What does this PR do?
Adds support for streamed responses for
chat.completions
and legacycompletions
calls. There are a few callouts for this PR:openai
versions>=4.1 <4.13
, the async iterator used for generating streamed responses returnsBuffer
objects from thenext
invocation. This is only an issue because some of these buffers contain half a chunk (iedata: {id: '...', choices: [{ delta: 'hi, this is a
, and following chunks contain the rest. Similarly, some chunks will also contain two object chunks in one buffer. To handle this, I wrote a small helper to take all of the buffers, concatenate them, and process them as specified in the OpenAI API documentation (which are just SSE, so they all start withdata:
, and the last element isdata: [DONE]
). Foropenai
versions>=4.13.0
, the elements returned from thenext
invocation are plain JSON objects.nock
was throughfs.createReadStream
with a data file. If there's a more efficient way of doing this, please let me know! But, I think this should be sufficient enough for the handful of tests this change needs.Motivation
Adds missing support for streamed responses for
chat.completions
and legacycompletions
.More OpenAI Follow-Ups
Plugin Checklist
Additional Notes