From c4d1738c0e0fce05cb5d98919e863f14c646b214 Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Fri, 17 May 2024 13:55:49 +0000 Subject: [PATCH] Speed up spellchecking by ignoring whitespace-only lines The new API has introduced extra overhead per line being spellchecked. One way of optimizing out this overhead, is to spellcheck fewer lines. An obvious choice here, is to optimize out empty and whitespace-only lines, since they will not have any typos at all (on account of not having any words). A side-effect of this change is that we now spellcheck lines with trailing whitespace stripped. Semantically, this gives the same result (per "whitespace never has typos"). Performance-wise, it is faster in theory because the strings are now shorter (since we were calling `.rstrip()` anyway). In pratice, I am not sure we are going to find any real corpus where the trailing whitespace is noteworthy from a performance point of view. On the performance corpus from #3491, this takes out ~0.4s of runtime brining us down to slightly above the 5.6s that made the baseline. --- codespell_lib/_codespell.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/codespell_lib/_codespell.py b/codespell_lib/_codespell.py index 8cbb1155b8..da9cbcafb6 100644 --- a/codespell_lib/_codespell.py +++ b/codespell_lib/_codespell.py @@ -948,7 +948,8 @@ def parse_file( ) for i, line in enumerate(lines): - if line.rstrip() in exclude_lines: + line = line.rstrip() + if not line or line in exclude_lines: continue extra_words_to_ignore = set()