make tokenizer emit at least one empty token on empty strings#5532

Open

trinity-1686a wants to merge 3 commits intomainfrom trinity/tokenizer-emit-empty-string

+167-8

Commits on Oct 30, 2024

make tokenizer emit at least one empty token
trinity-1686a
committed

Commits on Nov 6, 2024

only emit on strictly empty strings
trinity-1686a
committed

Commits on Nov 8, 2024

add integration test
trinity-1686a
committed