Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Community]: Ability to Passing a Tokenizer directly to TokenTextSplitter and updating from_tiktoken_encoder, from_huggingface_tokenizer #27036

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

[Community]: Ability to Passing a Tokenizer directly to TokenTextSplitter and updating from_tiktoken_encoder, from_huggingface_tokenizer #27036

wants to merge 7 commits into from

Commits on Oct 1, 2024

  1. [feat]: Added Ability in TokenTextSplitter to pass directly a instanc… 

    …e tokenizer
    @keenborder786
    keenborder786 committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    253a943 View commit details
    Browse the repository at this point in the history

  2. [chore]: Added Tests

    @keenborder786
    keenborder786 committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    d22c508 View commit details
    Browse the repository at this point in the history

  3. [chore]: Using tokenizer in and rather than length function which is … 

    …not being used by
    @keenborder786
    keenborder786 committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    615000e View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. [fix]: Should not remove the length function since they can be used w… 

    …ith other Splitter but add ability for Tokenizer to be passed for TokenTextSplitter
    @keenborder786
    keenborder786 committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    9e5e6f9 View commit details
    Browse the repository at this point in the history

  2. [chore]: More Tests Added

    @keenborder786
    keenborder786 committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    33675e5 View commit details
    Browse the repository at this point in the history

  3. [chore]: Linting issue in tests

    @keenborder786
    keenborder786 committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    a30e934 View commit details
    Browse the repository at this point in the history

  4. [chore]: Linting issue in tests

    @keenborder786
    keenborder786 committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    6888d30 View commit details
    Browse the repository at this point in the history