Fix custom dict error for unsupported tokenization engines #1066

wannaphong · 2025-01-12T13:23:10Z

Add error handling for unsupported custom dictionaries in word_tokenize function.

Add a check for unsupported engines in the word_tokenize function in pythainlp/tokenize/core.py.
Raise a NotImplementedError if custom_dict is passed to an unsupported engine such as attacut, icu, nercut, sefr_cut, tltk, and oskut.
Update the docstring for the word_tokenize function to reflect the changes.

Fixes #1065 Add error handling for unsupported custom dictionaries in `word_tokenize` function. * Add a check for unsupported engines in the `word_tokenize` function in `pythainlp/tokenize/core.py`. * Raise a `NotImplementedError` if `custom_dict` is passed to an unsupported engine such as `attacut`, `icu`, `nercut`, `sefr_cut`, `tltk`, and `oskut`. * Update the docstring for the `word_tokenize` function to reflect the changes.