how to tokenization 330ml or 330g

[jacentsao]

as below show, how to search 百事330

mysql> select name from product where match('百事330ml');
+------------------------------------------+
| name                                     |
+------------------------------------------+
| 百事可乐 罐装可乐型汽水 330ml            |
| 百事可乐可乐型汽水-长罐 330ml            |
| 百事可乐 罐装可乐型汽水 330ml            |
| 百事可乐 罐装可乐型汽水 330ml            |
+------------------------------------------+
4 rows in set (0.00 sec)

mysql> show meta;
+----------------+--------+
| Variable_name  | Value  |
+----------------+--------+
| total          | 4      |
| total_found    | 4      |
| total_relation | eq     |
| time           | 0.000  |
| keyword[0]     | 330ml  |
| docs[0]        | 30     |
| hits[0]        | 52     |
| keyword[1]     | 百事   |
| docs[1]        | 22     |
| hits[1]        | 22     |
+----------------+--------+
10 rows in set (0.00 sec)

mysql> select name from product where match('百事330');
Empty set (0.00 sec)

mysql> show meta;
+----------------+--------+
| Variable_name  | Value  |
+----------------+--------+
| total          | 0      |
| total_found    | 0      |
| total_relation | eq     |
| time           | 0.000  |
| keyword[0]     | 330    |
| docs[0]        | 1      |
| hits[0]        | 1      |
| keyword[1]     | 百事   |
| docs[1]        | 22     |
| hits[1]        | 22     |
+----------------+--------+
10 rows in set (0.00 sec)

[sanikolaev]

As discussed in Telegram, Manticore can't do it out of the box, but you can write a custom tokenizer plugin - https://manual.manticoresearch.com/Extensions/UDFs_and_Plugins/Plugins/Token_filter_plugins#Token-filter-plugins