Generate alternative typos with a translation table

[DimitriPapadopoulos]

This way we can catch misspellings with alternative characters, typically an acute accent (´) instead of an apostrophe ('). In this case, the alternative character is a valid character and will be used both in the misspelling and the fix(es).

The above is different from detecting Unicode phishing, where some characters like A are intentionally, or not, replaced by lookalikes such as A, Α, А, ᗅ, ᴀ, Ａ. In this case, the alternative character is invalid and should be replaced by its valid counterpart in the fix.

The latter use case is deferred to a different merge request.

Fixes #2984.

[DimitriPapadopoulos]

@larsoner This is a proof of concept, I will add tests if there is interest (typically to detect and avoid replaced characters in dictionaries).

[peternewman]

Seems much more sensible than my suggestion!

Microsoft "smart" quotes too?

Will also want some checks that we don't duplicate in the dictionary with different variants. Likewise if someone ignores with a different escape it will need to still match and remove it.

Presumably correction will have our deemed acceptable one?

How does this interact with the escape stuff someone did a little while ago?

[DimitriPapadopoulos]

By adding alternatives while reading the dictionaries, it really works as additional entries in the dictionary. Hence, we cannot, and I suggest we should not, ignore alternatives:

1. There might be niche cases where we want to ignore only the specified alternative.
2. In most cases, only a single alternative will be present in the checked files.
3. For the above reasons, it is not worth the additional complexity.

Duplicates will be avoided by checking all target characters are missing from the dictionaries. For example, all misspellings of acute accent (´) would have to be removed from the dictionaries, or the acute accent replaced by an apostrophe.

[DimitriPapadopoulos]

@peternewman Which escape stuff are you referring too?

I have not added smart single quotes because they are not part of the default word regex, which is probably used in the vast majority of cases:

codespell/codespell_lib/_codespell.py

Line 31 in 2eeb8b9

word_regex_def = "[\\w\\-'’`]+"

I could add smart single quotes (or possibly just the right single quotation mark)) to both the default word regex and my alternative to the apostrophe. But then, we don't want quotation marks to be part of parsed words, don't we? We'd miss plenty of misspellings at the end of quoted text.

Similarly, I have left out the grave accent because of #2981 and Markdown.

[larsoner]

Can you modify some clear/informative test that fails on master but passes on this PR? It will both act as a regression test and help reviewers understand the use case concretely...

[DimitriPapadopoulos]

The Windows error is related to encoding, obviously: According to the Python documentation, when writing:

encoding is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever locale.getencoding() returns),

Need to investigate how the test file is encoded when writing, then decoded when reading.

[DimitriPapadopoulos]

When reading files, the FlleOpener tries these encodings:

codespell/codespell_lib/_codespell.py

Line 39 in adf6ff6

encodings = ("utf-8", "iso-8859-1")

I will try to explicitly use the utf-8 encoding when writing, to match the first encoding utf-8 tried while reading.

Edit: It works!

[DimitriPapadopoulos]

@larsoner Can you have a look? Obviously, reading dictionaries will be slower, as I have added a function to process and sometimes duplicate dictionary lines.

[larsoner]

Seems reasonable in principle

Obviously, reading dictionaries will be slower

Looks like it is:

$ python -m timeit "import codespell_lib; codespell_lib.main('./codespell')"
10 loops, best of 5: 35.7 msec per loop
$ git checkout -b acute DimitriPapadopoulos/acute
$ python -m timeit "import codespell_lib; codespell_lib.main('./codespell')"
5 loops, best of 5: 42.1 msec per loop

But an extra 7ms is probably acceptable.

[larsoner]

Thanks @DimitriPapadopoulos