Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add characterization tests for basic langchain text responses and stream data. #676

Merged
merged 14 commits into from
Oct 26, 2023

Conversation

lgrammel
Copy link
Collaborator

@lgrammel lgrammel commented Oct 25, 2023

Overview

  • Introduce x-flush-delay header in mock server to speed up tests (default to 5ms to speed up existing tests)
  • Introduce port configuration for mock server to allow running several tests in parallel
  • Add langchain dev dependency
  • Add langchain characterization tests for existing behavior based on examples from website

Note

The langchain stream test does not work in the edge environment and is deactivated in that environment. The errror is caused by the following chain of events:

  • langchain functionality is required for langchain integration testing, so I added a dev dependency
  • langchain is used in other parts of the project, therefore the dependencies become non-dev dependencies in the lockfile
  • langchain has a dependency on uuidjs/uuid
  • uuidjs/uuid has an open bug with their dist: [BUG] Unexpected token 'export' with Jest (on Cloudflare Workers) uuidjs/uuid#678 (that’s open for almost a year)
  • the edge environment testing does not compile with

Details:

    /Users/lgrammel/repositories/vercel-ai/node_modules/.pnpm/[email protected]/node_modules/uuid/dist/esm-browser/index.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){export { default as v1 } from './v1.js';
                                                                                      ^^^^^^

    SyntaxError: Unexpected token 'export'

const response = new StreamingTextResponse(stream);

expect(await readAllChunks(response)).toEqual([
'',
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With Langchain, there is an empty text chunk at the beginning and the end. Is this desired?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but it shouldn't be a problem and not worth investigating until we finalize the complex mode API imo

@lgrammel lgrammel changed the title Add tests for basic langchain text responses and stream data. Add characterization tests for basic langchain text responses and stream data. Oct 25, 2023
@lgrammel
Copy link
Collaborator Author

The error is caused by the following chain of events:

  • langchain functionality is required for langchain integration testing, so I added a dev dependency
  • langchain is used in other parts of the project, therefore the dependencies become non-dev dependencies in the lockfile
  • langchain has a dependency on uuidjs/uuid
  • uuidjs/uuid has an open bug with their dist: [BUG] Unexpected token 'export' with Jest (on Cloudflare Workers) uuidjs/uuid#678 (that’s open for almost a year)
  • the edge environment testing does not compile with
    Details:

    /Users/lgrammel/repositories/vercel-ai/node_modules/.pnpm/[email protected]/node_modules/uuid/dist/esm-browser/index.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){export { default as v1 } from './v1.js';
                                                                                      ^^^^^^

    SyntaxError: Unexpected token 'export'

I’ve tried setting up Jest transformIgnorePatterns, but it does not help.

@lgrammel lgrammel changed the title Add characterization tests for basic langchain text responses and stream data. [WIP] Add characterization tests for basic langchain text responses and stream data. Oct 25, 2023
@lgrammel
Copy link
Collaborator Author

Tests are not run currently.

@lgrammel lgrammel changed the title [WIP] Add characterization tests for basic langchain text responses and stream data. Add characterization tests for basic langchain text responses and stream data. Oct 25, 2023
@lgrammel lgrammel merged commit 7f58d9b into vercel:main Oct 26, 2023
8 checks passed
@lgrammel lgrammel deleted the lg/langchain-tests branch October 27, 2023 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants