Consume any token except

[jonestristand]

Is there a method, or at least a canonical way, to consume any token EXCEPT a particular token (e.g. in a string, consume everything until hitting an end quote)?

[msujew]

@jonestristand You can use positive lookahead in your regex to lex up to a particular character/string. Something like /\$[\s\S]*?(?=\$)/ will parse everything starting from a $, up to (but not including) the next $.

[jonestristand]

Thanks for the reply - unfortunately in my case that would be pretty unwieldly, as I'd have to have every token written with a custom matcher. Basically I have wholly different sets of tokens that are available in each lexer mode.

In the future, a pop_all_modes property on the token would be useful and likely trivial to implement? Alternatively, am I able to access the internal lexer mode stack directly?

[bd82]

In the future, a pop_all_modes property on the token would be useful and likely trivial to implement? Alternatively, am I able to access the internal lexer mode stack directly?

That is unlikely.
The scenario described here seems like a bit of an edge case.
If your lexer requires advanced logic, beyond a a few individual tokens (via custom tokens).
Then it seems it may be a good idea to evaluate a completely custom lexer.

For example, in the ECMA5 example the Acorn lexer is used and Chevrotain tokens are constructed from the Acorn results.
That is because ECMA5 ambiguities in it's lexing rules which requires parsing context to resolve, so that is beyond the scope of Chevrotain.

In regards to accessing the internal lexer state, that is also unlikely to happen
because the lexer's tokenize method is extremely optimized for performance and simply not designed for such extensibility...

[jonestristand]

That is too bad! I have to say, Chevrotain is extremely cool, and the amount of effort and optimization that has gone into it is really impressive!

I don't know if it's as much as an edge case as you're saying, and it seems a shame to lose all of the optimization in the lexer by writing a custom lexer. Any situation with additional lexer modes that are optional to enter could wind up needing such a function. In my case, the lexer mode needs to be reset on each line, but you could imagine a similar situation if a design has included files concatenated together for lexing and wanted to reset to the default state at the start at each included file. The overhead and complication from having a 'pop_all_modes' flag on a token seems very trivial - take off all elements of the lexer mode stack and push back the same 'default_state' property provided in the lexer definition. It wouldn't even need to check the size of the lexer stack, since you'd be guaranteed to be left with a single element!

Thanks for considering!

[jonestristand]

It occurs to me I may be missing a better way of doing this anyway... In my case I am working on lexing/parsing the hledger journal syntax. So for example I have a line like this:

    Account:Chequing 1      -$ 5.00 @ 20 US:D ; tag_name_1: tag_val_1, tag_name2: tag_val_2

In this case, the line starts in a new lexer mode (for account posting lines). The colon in US:D is simply part of the commodity name, and in the comment (after the ;) separates tag names from values. The comment is optional though and may or may not exist. I enter the comment_lexer_mode on hitting the ;, but to get back to default_mode for the next line, I don't know if the NEWLINE token needs to pop 1 mode (just the posting line), or 2 (posting line, comment)... thoughts?

[bd82]

I am not familiar enough with the specific grammar/lexer here to contribute much.
But treating the colon as part of the the commodity name seems like a good idea.

Generally speaking if you can distinguish between these two kinds of colons you may not even need lexer modes:

• Are the separator colons always followed by whitespace? e.g: tag_name2: tag_val_2
• Are the commodity name colons never separated by whitespace from the rest of the commodity name?
• Do the commodity names never start with a colon?

Answering these questions may assist in distinguishing between these "kinds" of colons.