Parser Overhaul

[IamTheCarl]

Goals and reasoning

The current design of the parser produces unusable error messages and is awkward to extend.
A redesign is called for. It needs to support (or be easy to add) the following features:

• Continue parsing a file after the first error
• When multiple parse branches were tried, give context as to why they all failed, and not just why the most recent one failed.
• Move more errors from runtime to parsing
  • The unit type of scalar constants should immediately produce an error, rather than waiting until they are executed.

New design overview

The following layers should be used to process input files.

• Source String
  • Raw input from a file
  • Verify UTF8 encoding
• Tokenizer
  • Convert to tokens
    • Comments
    • Literals
    • Operators
    • Statement keywords
• Abstract Syntax Tree
  • Convert tokens into context sensitive objects
    • Struct definitions
    • Function definitions
    • Statements

This design should simplify the creation of higher level structures and the reporting of error messages related to them.
It should also make it much easier to write a formatter later.