Info about lexers

[safe60]

Hi. I'm learning how to write lexers for my project and until now I was able to write a simple lexer which easily recognize tokens.

My lexer should be fairly simple but it seems I lack knowledge about how to get my goal.

The lexer I'm going to write is for GCODE programs. Gcode is a language for computerized numerical control machines.
It's used worldwide since may be 50 years.

Here is a sample of a gcode program :

%
G40 G80 G49 G90
T4 M6 S2000
G0 X100 Y100 M03
G43 Z100H4
Z50
G01 Z10 F1500
X200 Y300
...
... thosands of lines ...
...
G0 Z100
Z500
M30
%

Every G followed by a number is a command of the language.

Interesting commands are G0 and G1 (accepted syntax is also G00 or G01).

G0 (or G00) tells machine to go in rapid mode.
G1 (or G01) tells machine to go in feed mode.

These two commands are the only one I'm interested in.
My goal would be :
If a line contains a G0 (or a G00) then style the whole line in let's say red color on white background
If a line contains a G1 (or a G01) then style the whole line in let's say green color on white background

The tricky thing is related to this :
both G0 and G1 are modal commands. A modal command will stay active until a different command is found.
In the sample above there is a G0 in the line just below the one which contains T4. This line should be colored in red to
indicate that rapid motions has begun.
Following lines should also be colored in red because rapid mode started by G0 token is still active.
Then comes the line with G01. This line should be colored in green to indicate feed mode is active.
all the subsequent lines of the file should also be marked in green.
When le the lexer find the last G0 it should revert to rapid mode.

I hope this is clear.

In my lexer I defined two categories of tokens :
one match G0 or G00 and assign to it a style with a red color.
The other category match G1 or G01 and assign to it a style with a green color.

First problem :
In a line which contains, for example, G0, only the token is marked in red. Other characters in the same line are colored with the default style.

Second problem :
if a line doesn't contains a G0 or a G1 the lexer assign to it the default style, probably because there are no rules which tell it what to do.
I need to "tell" the lexer to use as a default style the last one used.

For the first problem, after reading the docs, I thought that definining the lexer with the lex_by_line flag set to true would be helpful
but I don't know how to use this feature to lex all the line with the same style.

For the second problem... I hope it can be done and I'm accepting suggestions.

[orbitalquark]

1. If you go the "lex_by_line" route, you can simply match to `P(-1)` after each G token. See the *diff.lua* for an example of a line lexer. 2. I think you can use the `lexer.line_state` table to keep track of line states. You'd probably have an initial (or after the G rules) `P(function() ... end)` rule to check the state of the previous line and potentially match to the end of line in a particular G style. I don't have any experience with `line_state`, so this approach may not work too well. It might be worth simply defining a `lex:lex()` function in order to have complete control over the lexing process. This is not documented though. *lexer.lua* has a `lex()` function that all lexers inherit. You can use it as a reference.