Skip to content

Add tokenizer

Pre-release
Pre-release
Compare
Choose a tag to compare
@xhluca xhluca released this 08 Sep 05:00
· 26 commits to main since this release
0.2.0rc8
0a49c62
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

In this release, we add the Tokenizer class. Please see readme section on tokenization and examples/tokenizer_class.py for more details.

Assets 2
Loading