Skip to content
Back to pull request #27036

[Community]: Ability to Passing a Tokenizer directly to TokenTextSplitter and updating from_tiktoken_encoder, from_huggingface_tokenizer #30972

Sign in to view logs

[Community]: Ability to Passing a Tokenizer directly to TokenTextSplitter and updating from_tiktoken_encoder, from_huggingface_tokenizer

[Community]: Ability to Passing a Tokenizer directly to TokenTextSplitter and updating from_tiktoken_encoder, from_huggingface_tokenizer #30972

cd libs/text-splitters  /  make test # pydantic: ~=2.5.0, python: 3.11,

cd libs/text-splitters / make test # pydantic: ~=2.5.0, python: 3.11,
succeeded Oct 2, 2024 in 15s