adding tokenizer configuration

[jwnz]

Description

This PR aims to add some better functionality for configuration of the tokenizer (#5).

Todo

• Allow for selection of huggingface tokenizer. Downloads the model from the hub.
• Jieba tokenizer (chinese)
• tiktoken
• tiniestsegmenter (japanese) [optional]
• Allow switching of HF tokenizers
• Add Tests for tokenizing

Notes

1. Currently, if a HuggingFace tokenizer is initialized, the tokenizer cannot be changed. e.g. Doing SELECT tokenize('i have an apple', 'hf', 'sentence-transformers/LaBSE'); after SELECT tokenize('i have an apple', 'hf', 'google-t5/t5-base'); would just use the LaBSE tokenizer as it has already been initialized. Need a better way to handle this.

2. There is no official rust crate for tiktoken. Used tiktoken-rs

Usage

SELECT tokenize('i have an apple', 'hf', 'google-bert/bert-base-uncased');
SELECT tokenize('i have an apple', 'hf', 'google-t5/t5-base');
SELECT tokenize('i have an apple', 'tiktoken', 'gpt2');
SELECT tokenize('i have an apple', 'tiniestsegmenter', '');
SELECT tokenize('i have an apple', 'jieba', '');
SELECT tokenize('i have an apple', 'ws', '');

[VoVAllen]

We can use https://github.com/zurawiki/tiktoken-rs since it's just a wrapper. For more tokenizer support, @usamoi do you have any suggestion?

[VoVAllen]

Can you fix the DCO with the instruction?

[jwnz]

Signed off the commits. Apologies for the inconvenience.

[RobertHH-IS]

Will this be added? Critical to be able to adjust the tokenizer for multilingual support.

[VoVAllen]

We'll definitely merge this PR. However the maintainer is on the vacation now, will be back at 10/8.