ethereum-tests error messages

[ScottyPoi]

Continuing discussion from #3871.

Decide whether and how to use the expected error messages from ethereum-tests test vectors.

ethereum-tests include test instances meant to cause errors in execution, and include the type of error, or reason that it should fail, as part of the test vector. Currently, our test runners validate that an error has been thrown in these instances, but do not validate that the error thrown matches the expected error in the test.

One reason for this is that various optimizations result in errors being thrown as early as possible for invalid input, which can obscure the specific reason that the input was invalid.

For example, these two test instances use invalid input. Since we can invalidate the input based on size alone, we do not throw a specific error, but throw a size error:

expected = 'EOF_IncompleteSectionNumber'
actual = 'container size less than minimum'

and

expected = 'EOF_CodeSectionMissing'
actual = 'container size less than minimum'

In most cases, this mismatch will be the result of an optimization such as this. We want to preserve optimizations, and avoid altering our code to match a test vector. In rare cases, however, we may find ourselves "failing for the wrong reason", which would result in a test passing despite our code being incorrect.

Validating our errors against the test vectors is not as simple as applying a 1-to-1 mapping of expected errors to actual messages either, as we do end up with overlap in both directions.

--

One use case for these messages is to use them to inform when a test does fail. In these cases an error should have been thrown but wasn't. It would be helpful to debugging to have the expected error displayed when the test fails. i.e. Should have failed because of: ${_expected}

--

Todo:

• apply expected error messages to test failure logs
• design a way to validate that we are indeed passing "error" tests for the right reasons.