Token categories hide their sub-tokens in syntax diagrams

[lsq-ahaseeb]

Token categories are used to classify a bunch of tokens for ease of reference, among other things. I personally like classifying all binary operator tokens under one BinaryOp token, for instance, so that I only have to write one $.CONSUME() call in the parser as opposed to 10 alternations.

But in the generated syntax diagram the underlying tokens of a category are not shown. Instead the category token is shown as a terminal with NOT_APPLICABLE as its alt text. The expectation is that either the category is replaced with all its children as alternatives, or the category is considered a non-terminal with another railroad showing its sub-tokens, as if it were a production rule. Or some other way to show both that these sub-tokens exist and that they belong to this category.

For example, in the chevrotain playground for 'JSON Grammar and CST Output', True and False show up in the syntax diagram as they're supposed to with proper alt texts. Let's replace the tokens as such:

  const Boolean = createToken({ name: "Boolean", pattern: Lexer.NA });
  const True = createToken({ name: "True", pattern: /true/, categories: Boolean });
  const False = createToken({ name: "False", pattern: /false/, categories: Boolean });

and replace the True and False alterations in the parser with a single { ALT: () => $.CONSUME(Boolean) },. Let's add Boolean to jsonTokens as well.

As you can now see in the diagram, True and False are nowhere to be seen and Boolean, which could be just an internal implementation detail, takes their place in the diagram with an unhelpful alt text NOT_APPLICABLE.

There should be an option to allow expanding/collapsing categories or showing/not showing them as non-terminals.

[bd82]

Hello @lsq-ahaseeb

This is a known issue: #1137 which is up for grabs.
Lets continue the discussion there if you are interested in contributing a improvement to the diagrams rendering.

closing as duplicate