New lexers #375
Open
New lexers #375
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds a C lexer with a tool to output lexing results. The lexer is not hooked up to the rest of Elixir yet. Why: Currently, Elxir uses a simple, single-regex, perl based lexer. This approach mostly works, but it has a few issues: * Strings are not parsed correctly - there is a mistake in the regex that causes it to get confused by escapes in strings. * DTS identifers are not parsed correctly - some valid identifers are not recognized, others are split by otherwise allowed characters (mostly commas) * It barely distinguishes between languages. Comments in Kconfig files are not parsed correctly (and are not parseable by a simple regex). * Only identifiers are handled. Some parts of Elixir (filters, doccomments parser) could probably use a more detailed token stream. Right now, each of these parts contains a different regex-based lexer/parser. This mostly works, but again, it means the same functionality is reimplemented in many different parts of Elixir. * It does not recognize numbers, which means that numbers are looked up in the database during updates. * Keyword blocklist/allowlist is shared between all languages Some of these issues could be directly addressed in the regex itself, and some could be addressed somewhere in Elixir. But since there is a need for more sophisticated code analysis (ex. compatible filters, doccoments) and leaving such a crucial part of Elxir to a perl one-liner seems quite hacky, a decision to implement proper lexing, with a differnt lexer for each supported language was made. Libraries considered: * Pygments - Pygments lexers are good for code highlighting. It seems that as long the token stream results in expected identifiers being highlighted, it's good enough for Pygments. That is okay. Pygments lexers could be modified to provide a more reliable token stream, but the question is - does that help a typical Pygments user? Is it worth maintainers time? My assumption is that Pygments is not meant to be a general code analysis tool, but a code highlighter. It does that well, and extending it to a general lexer for all languages could be painful. * PLY - It seems that it's mostly meant for education purposes and isn't maintained anymore. It's not very ergonomic, for exapmle the interface requires each lexer to be in a different file. * pycparser - good for C, but does not support macros * Other parsing libraries - I'm quite sure that at this stage and for Elxir's purposes, we a flat token stream is wanted, not a full AST. Partial parsing could be done on the token stream later. But if more complete analysis is necessary, then it's probably beter to leave it to tools specific to that language (see ctags). Parser rules are also typically more complicated. Goals: * Good identifiers support - ex. not all DTS identifers are parsed correctly right now * Better comments support - KConfig help texts, GNU assmebler comments are not parsed well at all * Usable token stream that can be reassembled back into a file - some code analysis may require information about punctuation or comments, besides identifiers. It's also good to be sure that each character was considered, especially if code is meant to be modified. Notes: The lexers will never be perfect. Languages change, file extensions are confusing (.h can mean C, DTS or assembler). The main idea is to increase reliability of identifer references search, but achieving total correctness may require more work than it's worth. I picked an approach that should be, I hope, easy to understand, maintain, and allow sharing as much code as possible between different lexers.
This commit adds tests for the C lexer. The "architecture" may be controversial - test snippets are stored as strings. This has some drawbacks. * It's pretty ugly * Whitespace sensitive test cases may require extra care * Grepping may get even more annoying (it probably makes sense to skip all files starting with test_). An alternative would be to store each test case and result as a different file. I didn't go with that approach because of the following reasons: * It's harder to use the built-in Python testing framework * Test cases should be short, but it's annoying to navigate between many different small files * Making a readable test result format requres extra parsing work in Elixir. It's doable, but also annoying.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I did a lot of manual testing by comparing dumps of databases created with old and new lexers (with some help of manually written tools, as the diffs spanned hundreds of thousands of lines). I can't promise that all useful references will stay, and all useless references will go away, but I'm pretty sure that for the vast majority of cases this works. And it does fix the strings issue.
If I had to pick one thing to remove, it would be (most of) assembler comments special-casing. File extensions are a lie and I don't think anyone really knows what comment syntax is supported by which assembler/target architecture/version combination. Not all combinations are covered either way, Linux dropped a lot of niche architectures since 3.0.