Cannot match characters when using the pattern [\S]+

[BlackGlory]

This is very strange, aren’t [\S]+ and \S+ the same pattern? Did I miss something?

code:

import { createToken, Lexer } from 'chevrotain'

test('pass', () => {
  const Text = createToken({
    name: 'Text'
  , pattern: /\S+/
  })

  const MyLexer = new Lexer([Text])
  const result = MyLexer.tokenize('test')

  expect(result.errors.length).toBe(0) // ok
})

test('error', () => {
  const Text = createToken({
    name: 'Text'
  , pattern: /[\S]+/
  })

  const MyLexer = new Lexer([Text])
  const result = MyLexer.tokenize('test')

  expect(result.errors.length).toBe(0) // oops
})

test('RE', () => {
  expect(/^\S+$/.test('test')).toBe(true) // ok
  expect(/^[\S]+$/.test('test')).toBe(true) // ok
})

errors:

      [
        {
          offset: 0,
          line: 1,
          column: 1,
          length: 1,
          message: 'unexpected character: ->t<- at offset: 0, skipped 1 characters.'
        },
        {
          offset: 1,
          line: 1,
          column: 1,
          length: 1,
          message: 'unexpected character: ->e<- at offset: 1, skipped 1 characters.'
        },
        {
          offset: 2,
          line: 1,
          column: 1,
          length: 1,
          message: 'unexpected character: ->s<- at offset: 2, skipped 1 characters.'
        },
        {
          offset: 3,
          line: 1,
          column: 1,
          length: 1,
          message: 'unexpected character: ->t<- at offset: 3, skipped 1 characters.'
        }
      ]

[bd82]

Seems like a bug in RegExp to AST.

• https://github.com/bd82/regexp-to-ast/blob/master/lib/regexp-to-ast.js#L485

A whole characterSet can be marked as a complement, but the complement info of "negative" character escapes (\S, \D ...) are ignored.

[bd82]

I opened an issue in regExp to AST.

• Complement Character escapes inside a CharacterSet are not taken into account bd82/regexp-to-ast#118

But its not a high priority, I hope to have time to do some maintenance work on RegExpToAst and transfer it to this
organization (or even part of this mono-repo) next month when I should have some time off work 😄

[BlackGlory]

I am trying to avoid this bug with custom pattern, like this:

function regexp(regexp: RegExp) {
  const stickyRegexp = new RegExp(regexp, 'y')

  return function (text: string, startOffset: number) {
    stickyRegexp.lastIndex = startOffset
    const result = stickyRegexp.exec(text)
    return result
  }
}

const Text = createToken({
  name: 'Text'
, pattern: regexp(/[\S]+/)
, line_breaks: false
})

But it still prints Warning: No LINE_BREAKS Found., is this expected behavior?

[bd82]

That is a different issue, see:

• https://chevrotain.io/docs/guide/resolving_lexer_errors.html#LINE_BREAKS

[bd82]

You can disable the optimizations of the Lexer using safeMode which would workaround the bug.

• https://chevrotain.io/documentation/9_0_1/interfaces/ilexerconfig.html#safemode

You can also perform the computation needed for the optimization yourself by providing a start_chars_hint property, see:

• https://chevrotain.io/docs/guide/resolving_lexer_errors.html#CUSTOM_OPTIMIZE

[bd82]

The customToken is a valid workaround too, as it basically disables the optimizations in-directly as it is not possible to compute the start_chars for all the Tokens if you use a custom token pattern.