Ignore unreachable token error

[jonestristand]

I have a custom matcher that matches a regex pattern (e.g. "#") only if it is at the start of a line. Unfortunately I also need to match that token elsewhere (if not at start of line) and as a result get the unreachable error, since the lexer can't account for the custom logic in the matcher. Is there any way to disable this since I know that the other token will match sometimes?

[msujew]

@jonestristand I don't think that you can deactivate specific lexer validations, but that shouldn't be too much of an issue in development anyway. For production use of your lexer/parser it's either way recommended to disable all validations. See this documentation page.

[bd82]

Hello @jonestristand can you be more specific ?
Maybe provide a mini reproducible example?

[jonestristand]

Sure, here you go!

export function matchOnlyAtStart(regex: RegExp) {
  const matcher: CustomPatternMatcherFunc = (
    text,
    offset,
    tokens
  ): CustomPatternMatcherReturn | null => {
    const prevToken = _.last(tokens);
    if (
      offset === 0 ||
      prevToken?.tokenType.name === BasicTokens.NEWLINE.name
    ) {
      regex.lastIndex = offset;
      const match = regex.exec(text);
      return match ? [match[0]] : null;
    } else {
      return null;
    }
  };

  return matcher;
}

const ASTERISK = createToken({
  name: 'ASTERISK',
  pattern: '*'
});
const ASTERISK_AT_START = createToken({
  name: 'ASTERISK_AT_START',
  pattern: matchOnlyAtStart(/\*/y),
  line_breaks: false,
  start_chars_hint: ['*'],
  push_mode: 'comment_mode'
});
const tokens = [
  ASTERISK_AT_START,
  ASTERISK
];

const lexer = new Lexer(tokens);

[bd82]

This is the code I am running without error (on lexer init).

import { createToken, Lexer } from "chevrotain"

export function matchOnlyAtStart(regex) {
  const matcher = (text, offset, tokens) => {
    const prevToken = tokens[tokens.length - 1]
    if (
      offset === 0 ||
      prevToken?.tokenType.name === BasicTokens.NEWLINE.name
    ) {
      regex.lastIndex = offset
      const match = regex.exec(text)
      return match ? [match[0]] : null
    } else {
      return null
    }
  }

  return matcher
}

const ASTERISK = createToken({
  name: "ASTERISK",
  pattern: /\*/
})
const ASTERISK_AT_START = createToken({
  name: "ASTERISK_AT_START",
  pattern: matchOnlyAtStart(/\*/y),
  line_breaks: false
})
const tokens = [ASTERISK_AT_START, ASTERISK]

const lexer = new Lexer(tokens)

Do you get a different result?

[jonestristand]

I don't for asterisk, but do for semicolon and hashtag. Can you try it with those two?

edit: interesting: it works as desired if I use patterns /\#/ and /\;/ for HASHTAG and SEMICOLON and HASHTAG_AT_START and SEMICOLON_AT_START, but not if I use /#/ and /;/

[bd82]

interesting: it works as desired if I use patterns /#/ and /;/ for HASHTAG and SEMICOLON and HASHTAG_AT_START and SEMICOLON_AT_START, but not if I use /#/ and /;/

Yeah I just realized this as well.
There must be some validation / optimization around single character regexps somewhere... :)

[jonestristand]

Very interesting that it doesn't recognize escaped chars. I wonder if this points to a bug in the regexp library that chevrotain is using?

[bd82]

Very interesting that it doesn't recognize escaped chars. I wonder if this points to a bug in the regexp library that chevrotain is using?

I suspect is some edge case "best" attempt that when a regexp is only one char (excluding some special charts) it can safely be assumed to be converted to the matching string.

on a side note: The regexp library I wrote (regexp-to-ast) does have some missing features from new ECMAScript versions, perhaps I will have time during the upcoming holidays to update it and merge it into this mono-repo.

Cheers.