CPU usage at 100% on all cores #20
Comments
|
That does sound wrong. I take it flake8 and your other plugins worked fine? Is your small project online? That would be the easiest way for me to try to solve this. Can you share your flake8 and black configuration files, and the command line used to run flake8? |
|
Thanks, I'm 99.999% it's wrong configuration on my part - I'll get out it up later when I'm in front of that computer. |
|
If you can work out what was wrong, please do let me know - I might be able to guard against it. This is a very poor way for the plugin to fail (even if given bad input - which may not be the case). |
|
Hi, It's possible to trigger it, by running S |
|
Frustratingly it works for me, at least with my local copy of flake8 and its plugins: This could be down to the versions, and/or a conflict with another plugin. I used: Using |
|
Here is the output of I tried getting it to recreate the virtualenv and was able to reproduce it. The biggest difference is probably that I am running on Linux and you on OSX. ? {
"dependencies": [
{
"dependency": "entrypoints",
"version": "0.3"
}
],
"platform": {
"python_implementation": "CPython",
"python_version": "3.7.5",
"system": "Linux"
},
"plugins": [
{
"is_local": false,
"plugin": "black",
"version": "0.1.1"
},
{
"is_local": false,
"plugin": "flake8-annotations",
"version": "1.1.3"
},
{
"is_local": false,
"plugin": "flake8-bandit",
"version": "2.1.2"
},
{
"is_local": false,
"plugin": "flake8-bugbear",
"version": "20.1.3"
},
{
"is_local": false,
"plugin": "flake8-docstrings",
"version": "1.5.0, pydocstyle: 5.0.2"
},
{
"is_local": false,
"plugin": "import-order",
"version": "0.18.1"
},
{
"is_local": false,
"plugin": "mccabe",
"version": "0.6.1"
},
{
"is_local": false,
"plugin": "pycodestyle",
"version": "2.5.0"
},
{
"is_local": false,
"plugin": "pyflakes",
"version": "2.1.1"
}
],
"version": "3.7.9"
} |
|
(Updated - original comment has a typo in the Python version and was missing one plugin) Still on macOS, with Python 3.7.4, but with matching versions of flake8 and its plugins: That works for me. Time to try Linux.... |
|
Running out of ideas now, Linux with Python 3.7.6: Could be the exact flavour of Linux. Could be the exact version of Python 3.7. Neither seems likely. Could it be the filesystem? I have had flake8 stall on our cluster when trying to examine lots of files and the storage layer was under stress. Could you double check locally on a fresh clone and checkout in git (just in case there are some local files which are triggering this)? |
|
The verbose logging has highlighted that each worker loads the black settings. This macOS machine has 4 cores; so did this Linux machine - so it used four workers: Also tried it on Linux with lots of cores - enough for one worker per file: The fact that each worker loads the black configuration does not in itself seem to be a problem - although it would be nicer to do this once if possible. This made me notice your example looks to be on an 8 CPU machine (so would have 8 workers and load the black config 8 times), but it was checking over 4296 files(!) There seem to be a lot of stray Python files on your system - the cumulative run time could explain flake8 apparently getting stuck at 100% CPU. |
|
I'm not in front of that computer now, but had a quick look through the code on flake8-black on my phone, and can't see anything that could cause this. I'd like to add some logging (or just print statements), to see where flake8-black gets to and try again, got any ideas the best places I could put that ? Hm - I just noticed all the files, that is weird - I wonder where they could be coming from ? 4 cores / 8 threads, the modern way of counting is weird to me - amazing what you can get in a mid-range laptop now ! |
|
Writing to https://github.com/peterjc/flake8-black/blob/v0.1.1/flake8_black.py#L56 |
|
Turns out there was an old virtualenv in my root folder, along with some other stuff I wasn't excluding - hence all those files. Thanks for your patience, and pointing me in the right direction - also have a bunch more flake8 plugins to try now too. |
|
Thanks for your patience too - I'd have been rather frustrated if we hadn't been able to get to the bottom of this in the end 👍 |
|
I had a similar experience with nox when changing flake8 versions without regenerating the virtual environment. Is there a version-dependent cache? |
|
When run directly, black makes a cache of the results for the input files, so repeat runs of black are very fast. It does not via the flake8 plugin, see #26. This can matter if you are repeatedly checking all your files (or a complex large file). If you are asking about nox and if it caches the software for flake8, plugins, etc in virtual environments - I don't know. |
Hi,
I added flake8-black to poetry, it's been running for over a 5 minutes on my small project and 8 workers on 100% CPU - I'm guess this is not expected behaviour ?
So - I guess my config is wrong and it's stuck in a loop - this is probably something that still shouldn't be possible though ?
The text was updated successfully, but these errors were encountered: