Skip to content

Tokenizer 1.26.0
Compare
Choose a tag to compare
@guillaumekln guillaumekln released this 19 Apr 08:28
· 97 commits to master since this release
v1.26.0
590330f

New features

  • Add lang tokenization option to apply language-specific case mappings

Fixes and improvements

  • Use ICU to convert strings to Unicode values instead of a custom implementation
Assets 2
Loading