UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 0: invalid start byte

[sentry-io]

Sentry Issue: AUTHOR-TOOLS-36K

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 0: invalid start byte
(8 additional frame(s) were not displayed)
...
  File "at/utils/authentication.py", line 37, in require_api_key
    return f(*args, **kwargs)
  File "at/utils/file.py", line 178, in check_file
    return f(*args, **kwargs)
  File "at/api.py", line 48, in render
    dir_path, filename = process_file(
  File "at/utils/processor.py", line 44, in process_file
    filename = md2xml(filename, logger)
  File "at/utils/processor.py", line 54, in md2xml
    first_line = file.readline().strip()

[kesara]

Related to #469

[cabo]

Right, I get

SyntaxError: Unexpected token '<', "

still when I try the slightly broken document that started #469 (and that doesn't have anything suspicious in the first 6915 characters).

[cabo]

So there seems to be some processing in the authortools that tries to make use of the UTF-8-ness of the data before handing over the data to kramdown-rfc

[kesara]

Yes, Author Tools read the first line of a markdown file to identify which markdown tool to use.

[cabo]

I see. You could do this with the raw bytes instead of requiring the whole file to be proper UTF-8.