Fuzzer Discovery: issue relating to multi-line strings and newlines at the EOF

[jayaddison]

Describe the bug
The fuzz.py script discovered the following error during this build:

Traceback (most recent call last):
  File "/home/runner/work/black/black/fuzz.py", line 60, in <module>
    test_idempotent_any_syntatically_valid_python()
  File "/home/runner/work/black/black/fuzz.py", line 17, in test_idempotent_any_syntatically_valid_python
    max_examples=1000,  # roughly 1k tests/minute, or half that under coverage
  File "/home/runner/work/black/black/.tox/fuzz/lib/python3.9/site-packages/hypothesis/core.py", line 1158, in wrapped_test
    raise the_error_hypothesis_found
  File "/home/runner/work/black/black/fuzz.py", line 43, in test_idempotent_any_syntatically_valid_python
    dst_contents = black.format_str(src_contents, mode=mode)
  File "/home/runner/work/black/black/src/black/__init__.py", line 1025, in format_str
    src_node = lib2to3_parse(src_contents.lstrip(), mode.target_versions)
  File "/home/runner/work/black/black/src/black/__init__.py", line 1121, in lib2to3_parse
    result = drv.parse_string(src_txt, True)
  File "/home/runner/work/black/black/src/blib2to3/pgen2/driver.py", line 142, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/home/runner/work/black/black/src/blib2to3/pgen2/driver.py", line 70, in parse_tokens
    for quintuple in tokens:
  File "/home/runner/work/black/black/src/blib2to3/pgen2/tokenize.py", line 542, in generate_tokens
    raise TokenError("EOF in multi-line statement", (lnum, 0))
blib2to3.pgen2.tokenize.TokenError: ('EOF in multi-line statement', (2, 0))
Highest target scores:
               8  (label='(hypothesmith from_node) number of unique ast node types')
              22  (label='(hypothesmith) number of unique ast node types')
              63  (label='(hypothesmith) instructions in bytecode')
             114  (label='(hypothesmith) total number of ast nodes')
             172  (label='(hypothesmith from_node) instructions in bytecode')
             173  (label='(hypothesmith from_node) total number of ast nodes')

Falsifying example: test_idempotent_any_syntatically_valid_python(
    src_contents='\n\\\r\n',
    mode=Mode(target_versions=set(), line_length=88, string_normalization=False, magic_trailing_comma=True, experimental_string_processing=False, is_pyi=False),
)
ERROR: InvocationError for command /home/runner/work/black/black/.tox/fuzz/bin/coverage run fuzz.py (exited with code 1)

Expected behavior
The code that black can format should correspond one-to-one with valid Python language code.

Environment (please complete the following information):

• Version: 543fc7e
• OS and Python version: [e.g. Linux/Python 3.9]

Does this bug also happen on master?
Yes, most likely.

Additional context Add any other context about the problem here.
The newline at the end of the string may be important. When running the python interpreter on a file from the filesystem, the presence of a newline at the end of the file allows the multiline string indicator (\ character) to complete tokenization.

[ichard26]

I think this is a duplicate of #1012, but I'm not 100% sure. @jayaddison what do you think? Also, thanks for reporting!

[jayaddison]

D'oh! - it is, yep. Thanks @ichard26!