Load custom plugins when linting in parallel #8683
Conversation
jacobtylerwalls
changed the title
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #8683 +/- ##
==========================================
- Coverage 95.80% 95.80% -0.01%
==========================================
Files 172 172
Lines 18301 18300 -1
==========================================
- Hits 17534 17533 -1
Misses 767 767
|
|
π€ According to the primer, this change has no effect on the checked open source code. π€π This comment was generated for commit cb7804c |
| There is one known limitation with running checks in parallel as currently | ||
| implemented. Since the division of files into worker processes is indeterminate, | ||
| checkers that depend on comparing multiple files (e.g. ``cyclic-import`` | ||
| and ``duplicate-code``) can produce indeterminate results. |
There was a problem hiding this comment.
We have a map reduce function for that I think. But some are not implemented (they should). For duplicate code I don't even know if it's possible to parralelize. The whole hashmap of lines should be shared.
| @@ -30,6 +31,9 @@ | |||
| from pylint.typing import FileItem | |||
| from pylint.utils import LinterStats, ModuleStats | |||
|
|
|||
| if TYPE_CHECKING: | |||
There was a problem hiding this comment.
Not sure this helps much? If you are running this test then nodes will have probably been imported somewhere right?
There was a problem hiding this comment.
I think we shouldn't reject change because it's only a small improvments. It's better, let's ship it.
There was a problem hiding this comment.
Fair, I just prefer this conceptually, when it comes in handy in more substantial cases. We can probably trace this to the opposite views we expressed on #8111 :-)
|
Thanks for the reviews! |
Type of Changes
Description
Load custom plugins when processing in parallel.
The current implementation of multiprocessing launches a pool, and unpickles a
PyLinterobject that contains all the configuration information. The workers in the pool will not have access to the AstroidManager that had the configuration options (custom plugins, init hooks) applied to it.The solution here is to just re-register the custom plugins during the initialization of the pool. It's inelegant to do this multiple times, but I'm suggesting not over-optimizing a multiprocessing design that's flawed in other ways. (See #2525.) We can optimize this infelicity out later if it's still even relevant after the overhaul for #2525.
(It's possible some custom code in
--init-hookstill might not execute, but the main use case for--init-hookis for manipulating sys.path so that custom plugins will work, and they do work now, so I removed the caveat in the docs. No one on the issue board has presented another use case for--init-hookand--jobs.)Closes #4874
Closes #3232
The MRE in #4874 passes now π