Adding pytest-cov dependency to project breaks Pants commands
#21851
Comments
|
If you preserve the sandbox (ex with |
|
@cburroughs where in the sandbox would I find that? The sandbox contents look like: total 8064
drwxr-xr-x 9 ruslan staff 288 Jan 17 12:25 .
drwx------@ 423 ruslan staff 13536 Jan 17 12:25 ..
drwxr-xr-x 3 ruslan staff 96 Jan 17 12:25 .cache
lrwxr-xr-x 1 ruslan staff 63 Jan 17 12:25 .python-build-standalone -> /Users/ruslan/.cache/pants/named_caches/python_build_standalone
drwxr-xr-x 2 ruslan staff 64 Jan 17 12:25 .tmp
-rwxr-xr-x 1 ruslan staff 2117 Jan 17 12:25 __run.sh
drwxr-xr-x 3 ruslan staff 96 Jan 17 12:25 build-support
-r-xr-xr-x 1 ruslan staff 4124530 Jan 17 12:24 pex
drwxr-xr-x 2 ruslan staff 64 Jan 17 12:25 source_files
|
|
I'm attaching the stack trace and debug log when running |
|
Not sure if it's relevant, but running I have 3.10.15, which as previously mentioned is also what |
|
Within the sandbox The |
|
@ruslan-roboto, in your debug output: Clearly suspicious. Sure enough, I find on my machine: :; pyenv install --list | grep 3.10-dev
3.10-dev
cinder-3.10-dev
:; pyenv install 3.10-dev
Cloning https://github.com/python/cpython...
Installing Python-3.10-dev...
Installed Python-3.10-dev to /home/jsirois/.pyenv/versions/3.10-dev
# ~/dev/pex-tool/pex on sh-boot/more:main [$?⇡] via v3.13.1 took 1m28s
:; /home/jsirois/.pyenv/versions/3.10-dev/bin/python3.10 -V
Python 3.10.16+
Yes. So "Adding pytest-cov dependency to project..." seems to have ~nothing to do with this. It may have caused some sort of shuffle that exposed the 3.10.16+ Python interpreter version issue, but that's surely the actual issue here. Now, checking Pex (I maintain Pex so this issue caught my eye as a potential bug in Pex code): So current Pex handles this fine. pants/src/python/pants/backend/python/util_rules/pex_cli.py Lines 36 to 41 in 9a993cd So, trying that: So, as far as I can tell, this would seem to rule out Pex handling 3.10.16+ badly here. What I'm missing here from your setup is |
|
@jsirois Here's the contents of #!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /private/var/folders/7c/3ny04jh95k9fmjxtksxqyh2m0000gn/T/pants-sandbox-cRuNR7
env -i CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= PATH=$'/opt/homebrew/opt/make/libexec/gnubin:/Users/ruslan/.pyenv/shims:/Users/ruslan/.nvm/versions/node/v22.9.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/ruslan/.local/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON=/Users/ruslan/Library/Caches/nce/42dd077dd7c2238e60a71f8ccd2c52fce17bd09daa7b30d19c26707cc4443668/bindings/venvs/2.21.0/bin/python3.9 PEX_ROOT=.cache/pex_root /Users/ruslan/Library/Caches/nce/42dd077dd7c2238e60a71f8ccd2c52fce17bd09daa7b30d19c26707cc4443668/bindings/venvs/2.21.0/bin/python3.9 ./pex --tmpdir .tmp --jobs 8 --no-emit-warnings --pip-version 24.0 --python-path $'/Users/ruslan/.pyenv/versions/3.10-dev/bin:/Users/ruslan/.pyenv/versions/3.10.15/bin:/Users/ruslan/.pyenv/versions/3.10.9/bin:/Users/ruslan/.pyenv/versions/3.12.6/bin:/opt/homebrew/opt/make/libexec/gnubin:/Users/ruslan/.pyenv/shims:/Users/ruslan/.nvm/versions/node/v22.9.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/ruslan/.local/bin' --output-file black.pex --venv prepend --seed verbose --venv-site-packages-copies --python /Users/ruslan/.pyenv/versions/3.10-dev/bin/python3.10 --console-script black --no-pre-install-wheels $'--sources-directory=source_files' --lock build-support/lockfiles/python-default.lock --no-pypi $'--index=https://pypi.org/simple/' --manylinux manylinux2014 --layout packedAs for the lockfile, here's the section about {
"artifacts": [
{
"algorithm": "sha256",
"hash": "3bb2b7a1f7b685f85b11fed1ef10f8a9148bceb49853e47a294a3dd963c1dd7d",
"url": "https://files.pythonhosted.org/packages/8d/a7/4b27c50537ebca8bec139b872861f9d2bf501c5ec51fcf897cb924d9e264/black-24.10.0-py3-none-any.whl"
},
{
"algorithm": "sha256",
"hash": "649fff99a20bd06c6f727d2a27f401331dc0cc861fb69cde910fe95b01b5928f",
"url": "https://files.pythonhosted.org/packages/6e/c5/9023b7673904a5188f9be81f5e129fff69f51f5515655fbd1d5a4e80a47b/black-24.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl"
},
{
"algorithm": "sha256",
"hash": "e6668650ea4b685440857138e5fe40cde4d652633b1bdffc62933d0db4ed9812",
"url": "https://files.pythonhosted.org/packages/a3/f3/465c0eb5cddf7dbbfe1fecd9b875d1dcf51b88923cd2c1d7e9ab95c6336b/black-24.10.0-cp310-cp310-macosx_10_9_x86_64.whl"
},
{
"algorithm": "sha256",
"hash": "846ea64c97afe3bc677b761787993be4991810ecc7a4a937816dd6bddedc4875",
"url": "https://files.pythonhosted.org/packages/d8/0d/cc2fb42b8c50d80143221515dd7e4766995bd07c56c9a3ed30baf080b6dc/black-24.10.0.tar.gz"
},
{
"algorithm": "sha256",
"hash": "1c536fcf674217e87b8cc3657b81809d3c085d7bf3ef262ead700da345bfa6ea",
"url": "https://files.pythonhosted.org/packages/df/57/b6d2da7d200773fdfcc224ffb87052cf283cec4d7102fab450b4a05996d8/black-24.10.0-cp310-cp310-macosx_11_0_arm64.whl"
}
],
"project_name": "black",
"requires_dists": [
"aiohttp>=3.10; extra == \"d\"",
"click>=8.0.0",
"colorama>=0.4.3; extra == \"colorama\"",
"ipython>=7.8.0; extra == \"jupyter\"",
"mypy-extensions>=0.4.3",
"packaging>=22.0",
"pathspec>=0.9.0",
"platformdirs>=2",
"tokenize-rt>=3.2.0; extra == \"jupyter\"",
"tomli>=1.1.0; python_version < \"3.11\"",
"typing-extensions>=4.0.1; python_version < \"3.11\"",
"uvloop>=0.15.2; extra == \"uvloop\""
],
"requires_python": ">=3.9",
"version": "24.10.0"
},Thank you for your analysis so far! There are a couple of things I don't understand:
|
|
Note: I'm using 12:53:22.44 [INFO] Completed: Building mypy.pex
12:53:22.44 [ERROR] 1 Exception encountered:
Engine traceback:
in `check` goal
ProcessExecutionFailure: Process 'Building mypy.pex' failed with exit code 1.
stdout:
stderr:
Invalid version: '3.10.16+'The script for that is: #!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /private/var/folders/7c/3ny04jh95k9fmjxtksxqyh2m0000gn/T/pants-sandbox-bjpk4I
env -i CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= PATH=$'/opt/homebrew/opt/make/libexec/gnubin:/Users/ruslan/.pyenv/shims:/Users/ruslan/.nvm/versions/node/v22.9.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/ruslan/.local/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON=/Users/ruslan/Library/Caches/nce/42dd077dd7c2238e60a71f8ccd2c52fce17bd09daa7b30d19c26707cc4443668/bindings/venvs/2.21.0/bin/python3.9 PEX_ROOT=.cache/pex_root /Users/ruslan/Library/Caches/nce/42dd077dd7c2238e60a71f8ccd2c52fce17bd09daa7b30d19c26707cc4443668/bindings/venvs/2.21.0/bin/python3.9 ./pex --tmpdir .tmp --jobs 4 --no-emit-warnings --pip-version 24.0 --python-path $'/Users/ruslan/.pyenv/versions/3.10-dev/bin:/Users/ruslan/.pyenv/versions/3.10.15/bin:/Users/ruslan/.pyenv/versions/3.10.9/bin:/Users/ruslan/.pyenv/versions/3.12.6/bin:/opt/homebrew/opt/make/libexec/gnubin:/Users/ruslan/.pyenv/shims:/Users/ruslan/.nvm/versions/node/v22.9.0/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/ruslan/.local/bin' --output-file mypy.pex --venv prepend --seed verbose --venv-site-packages-copies --python /Users/ruslan/.pyenv/versions/3.10-dev/bin/python3.10 --console-script mypy --no-pre-install-wheels $'--sources-directory=source_files' --lock build-support/lockfiles/python-default.lock --no-pypi $'--index=https://pypi.org/simple/' --manylinux manylinux2014 --layout packed |
|
@ruslan-roboto you must provide the whole lockfile to repro, or at least a complete subgraph. You only supplied one locked requirement entry, that of black. You included 0 of its dependencies. The complete subgraph is needed. But in that path lies madness, please just supply your lockfile instead. It's hard to imagine a lock file is proprietary. Without a lockfile, the best I can do is like so: # Prove I'm using the same Pex PEX (Downloaded from here: https://github.com/pex-tool/pex/releases/download/v2.3.1/pex):
:; python3.9 ~/Downloads/pex -V
2.3.1
# Run the ~same command:
:; rm -rf ~/.pex ~/.cache/pex && env -i CPPFLAGS= LANG=en_US.UTF-8 LDFLAGS= PATH=/home/jsirois/.pyenv/shims:$PATH PEX_IGNORE_RCFILES=true PEX_PYTHON=python3.9 PEX_ROOT=.cache/pex_root python3.9 ~/Downloads/pex --tmpdir .tmp --jobs 4 --no-emit-warnings --pip-version 24.0 --python-path /home/jsirois/.pyenv/versions/3.10-dev/bin:/home/jsirois/.pyenv/shims --output-file mypy.pex --venv prepend --seed verbose --venv-site-packages-copies --python /home/jsirois/.pyenv/versions/3.10-dev/bin/python3.10 mypy --console-script mypy --no-pre-install-wheels --manylinux manylinux2014 --layout packed
{"pex_root": "/home/jsirois/dev/pex-tool/pex/.cache/pex_root", "python": "/home/jsirois/.pyenv/versions/3.10-dev/bin/python3.10", "pex": "/home/jsirois/dev/pex-tool/pex/.cache/pex_root/venvs/1bd9a579f43403aaed22d4101d48391a6c37e0fc/8902948047df79eb01bd149fc48673ac6cb58184/pex"}
What makes you think Pants respects
I have absolutely no idea why Going back to the OP:
@ruslan-roboto is it also the case that for every Mac laptop with the issue it also has 3.10-dev (3.10.16+) installed in pyenv? Can you confirm uninstalling 3.10-dev fixes ones of these machines? Etc., basically run through the obvious permutations to make this process of whittling down more efficient. @ruslan-roboto unless you can provide a real whole lockfile, I've reproed in as much detail as possible and Pex continues to be ruled out. I think you should try to pull a Pants person - perhaps @cburroughs - closer in to the problem since I'm pretty confident now its not a Pex problem and I maintain just Pex, I do not work on Pants. |
|
Here's a scoped-down lockfile which contains the versions of our tools dependencies - I found that it's sufficient for a repro (file extension changed to
It looks like the engineer who did not face this issue on a Mac also did not have |
|
@ruslan-roboto I don't repro with that lock file (after removing the @#$%! invalid-JSON header Pants puts on it): I will leave you to the Pants people, but remind you to answer my question about |
|
More attempts this am to repro. Still nothing. Here's my efforts for posterity: https://github.com/jsirois/pants-issue-21851 |
|
Hi @jsirois , I was able to repro the issue with your Github repository after making two changes:
I suspect that the first change is the important one. EDIT: confirmed - all you need to do for a repro is remove Thanks, |
|
@ruslan-roboto thanks for helping narrow down the repro - I also now repro. The underlying issue is here (Pex uses packaging 23.1 where the issue is not fixed): pypa/packaging#678 So, the Minimal repro: I'll update more after I figure out how best to get at the fix in packaging 24.1. That said, it continues to be very frustrating that you have not answered the |
|
Hi @jsirois , Thanks for pinning down the root cause! As for ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython==3.10.*' failed with exit code 102.
stdout:
stderr:
Could not find a compatible interpreter.
Examined the following working interpreters:
1.) /Users/ruslan/Library/Caches/nce/bf0cd90204a2cc6da48cae1e4b32f48c9f7031fbe1238c5972104ccb0155d368/cpython-3.9.18+20240107-aarch64-apple-darwin-install_only.tar.gz/python/bin/python3.9 CPython==3.9.18
2.) /opt/homebrew/Cellar/[email protected]/3.13.1/Frameworks/Python.framework/Versions/3.13/bin/python3.13 CPython==3.13.1
3.) /opt/homebrew/Cellar/[email protected]/3.12.6/Frameworks/Python.framework/Versions/3.12/bin/python3.12 CPython==3.12.6
4.) /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3.9 CPython==3.9.6
Skipped the following broken interpreters:
1.) /Users/ruslan/.pyenv/shims/python:
pyenv: python: command not found
The `python' command exists in these Python versions:
3.10-dev
3.10.15
3.12.6
Note: See 'pyenv help global' for tips on allowing both
python2 and python3 to be found.
2.) /Users/ruslan/.pyenv/shims/python3.10:
pyenv: python3.10: command not found
The `python3.10' command exists in these Python versions:
3.10-dev
3.10.15
Note: See 'pyenv help global' for tips on allowing both
python2 and python3 to be found.
(See https://github.com/pex-tool/pex/issues/1027 for a list of known breaks and workarounds.)
No working interpreter compatible with the requested constraints was found:
Version matches CPython==3.10.*Running Thanks, |
|
@ruslan-roboto the fix is available in Pex 2.23.1: https://github.com/pex-tool/pex/releases/tag/v2.32.1 Instructions for upgrading on your own so you don't have to wait for a Pants release are here: https://www.pantsbuild.org/stable/docs/python/overview/pex#setting-pex-and-pip-versions |
|
Thanks @jsirois ! |
``` Lockfile diff: 3rdparty/python/user_reqs.lock [python-default] == Upgraded dependencies == certifi 2024.12.14 --> 2025.1.31 ``` fixes pantsbuild#21851
Describe the bug
We recently added
pytest-covto our repo to measure test coverage. For some engineers on the team working specifically on Mac laptops, this causes a 100%-repro failure when running any Pants command, e.g.pants fmt :::Other issues on the Internet indicate that
3.10.16+is an untagged CPython version string, but rerunning the command with--print-stacktrace -ldebugdidn't indicate to me where this version gets picked up from. Our project uses pyenv, and.python_versionis set to3.10.python --versionreturns3.10.15for me.I've tried:
--no-pantsdand--no-local-cache- no effectpytestandpytest-cov- no effect, but whenever I regenerate our lockfile, thepytest-covdependency is shown in red in the update summary, and I don't know what that meansPants version
From our
pants.toml:The issue repros if I change this to 2.22 or 2.23, haven't tried 2.24.
OS
This repros on MacOS, but not on Linux. Also anecdotally not on all MacOS laptops within the team. I'm on MacOS Sonoma 14.7
Additional info
Add any other information about the problem here, such as attachments or links to gists, if relevant.
Output of
python -VVrun from the root of our repo:The text was updated successfully, but these errors were encountered: