Need clarity on python-tag #322
Comments
|
It's worth noting the FAQ entry "What tag do I use if my distribution uses a feature exclusive to the newest version of Python?" which pretty clearly states that a The fact that pip doesn't choose installation candidates using I just commented on jaraco/zipp#42, but it seems to be saying that a Python 3.7 installation cannot install a But I'm a strong -1 on seeing tags like |
I think we determined this issue was when running against DevPi, so with your clarification above, it should be possible to get DevPi to honor this intention. Thanks also for the clarification on multi-version tags. That aligns with my instinct. |
|
The purpose of the tags is to be able to choose the correct wheel when there can be multiple variants of a wheel for that package. For example, a different version for x86 or for ARM. Or if the source code runs through a transpiler to produce a different version for Python 2 and Python 3. This primarily works in the negative by eliminating all the wheels that definitely won't run on my machine, e.g. all the ones for different architectures; the ranking algorithm chooses if there are still multiple wheels after that.
Suppose you have a run of the mill pure Python wheel that has exactly one variant. In that case the tag's only job is to be accepted, because the installer doesn't have to choose the best wheel among several. It's not supposed to guarantee compatibility with every Python interpreter that accepts that particular tag.
…On Mon, Feb 17, 2020, at 11:27 AM, Jason R. Coombs wrote:
Closed #322 <#322>.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#322?email_source=notifications&email_token=AABSZERUA7UMCSHGUU2LB5LRDK3HFA5CNFSM4KWT4UR2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOWV2VYMA#event-3044367408>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABSZEUVM26GG73RKUKRAC3RDK3HFANCNFSM4KWT4URQ>.
|
|
Good timing on this thread- I have just built my first wheel and am confused about the default wheel behaviour. My scenario is quite straight forward, pure python, no python_requires defined, and not supplied any --python-tag param to bdist_wheel, and yet my resultant wheel defined it as suitable for py3, despite my source obviously requiring a more recent interpreter. Looking at the source for wheel, I can see this is the intended behaviour, however it has broken my expectations, and potentially any recepients of the package without the correct interpreter version. It should be either mandatory to provide the python tag, via whatever mechanisms are already available (python_requires, --python-tag, etc) or to do a better job of defaulting to something sensible using a more advanced mechanism than the major version of the interpreter the package was built by. Without either of these in place, installing a wheel which has defaulted to py3 is basically a gamble, it might work, it might not. In my case I have f-strings in the source, so require 3.6+. Is there any appetite for improving this, I am thinking something involving static src analysis. |
|
The wheel tag doesn't work that way. The negative match "this wheel doesn't work on my system", in the case of alternative wheels for a single version number, is what matters. For example your wheel could also contain a bug that prevents compatibility with any python, we wouldn't give that wheel a special tag.
If you decide to release two variants of the same version number of your software, one with python 3.6 compat and the other with python 3.7 compat, then the wheel tag is there for you. Otherwise it should be py3.
…On Mon, Feb 24, 2020, at 6:32 PM, c1432666 wrote:
Good timing on this thread- I have just built my first wheel and am confused about the default wheel behaviour. My scenario is quite straight forward, pure python, no python_requires defined, and not supplied any --python-tag param to bdist_wheel, and yet my resultant wheel defined it as suitable for py3, despite my source obviously requiring a more recent interpreter.
Looking at the source for wheel, I can see this is the intended behaviour, however it has broken my expectations, and potentially any recepients of the package without the correct interpreter version. It should be either mandatory to provide the python tag, via whatever mechanisms are already available (python_requires, --python-tag, etc) or to do a better job of defaulting to something sensible using a more advanced mechanism than the major version of the interpreter the package was built by.
Without either of these in place, installing a wheel which has defaulted to py3 is basically a gamble, it might work, it might not. In my case I have f-strings in the source, so require 3.6+.
Is there any appetite for improving this, I am thinking something involving static src analysis.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#322?email_source=notifications&email_token=AABSZES4KINKV7B77APVP6DRERKHBA5CNFSM4KWT4UR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMZ6ATI#issuecomment-590602317>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABSZEWPMHPW4ZFBINAXRBTRERKHBANCNFSM4KWT4URQ>.
|
|
Daniel, thanks for the prompt response. What you are saying makes sense, particularly when the context is many packages with lots of overlapping compatibility. However it does not diminish the fact the tag still indicates compatibility, negative selection or otherwise. If I build a wheel with compiled code for a specific architecture, the wheel gets tagged as such and only those running that architecture can expect it to work on their system. For the same reason, I would expect a wheel built containing interpreter version specific features to be identified as such. My point is, I have a very trivial use case, probably one which is common to vast numbers of pure python packages, and yet the tooling doesn't do the best possible job of supporting a working result. |
|
@c1432666 The wheel tags don't replace the "Python-Requires" metadata field. |
|
Right, now we're cooking. So in my trivial, yet common scenario, python requires is not mandatory, nor is a --python-tag, nor is any other method which stipulates the required interpreter version. This, and this alone leads to a suboptimal tag as you need to default to something right.... |
|
There's no way to determine without assistance what version of Python your code will support. You could have if sys.version_info <= (3, 7):
raise RuntimeErrorThe default assumes that unless told otherwise, there is no reason to think that the wheel won't work on any version of Python (that has the same major version as the wheel was built on). If that assumption isn't valid, just tell the tools that information. |
|
Hi Paul, thanks for your input.
Not true, static analysis can do this.
This is the crux of my point, this assumption is potentially harmful to a working installation of a wheel.
This is my secondary point, telling the tool something it could determine for you is less than ideal. Worse than that, it isn't mandatory so people like me, who didn't know any better get lulled into a false sense of security thinking everything is rosey and all is well. However not everyone is likely to spot this default assumption, go off investigate and understand the behaviour and it's repercussions. Outcome == poor consumer experience of said wheels. |
|
It's a contrived example, but if you applied the same logic to the platform tag, for a package with some compiled code and instead of setting the specific target arch, defaulting to any because it hasnt been provided by the packager. It might work for folks as most folks are running win64 or linux amd64 so you got a good chance, but by defaulting to a tag which is inaccurate introduces a potential barrier to its use. |
|
I'm not sure what you want to achieve here. In terms of the topic of this issue ("Need clarity on python-tag") I hope that's addressed - the current behaviour has been explained. You're arguing that the advice given in the PEP 425 FAQ "What tag do I use if my distribution uses a feature exclusive to the newest version of Python?" is wrong, and should be changed. If you want to do that, you should propose a change to that PEP, which would need to be agreed on the Discourse packaging group. Ultimately, though, you seem to be misunderstanding the purpose of compatibility tags, which is to choose the most appropriate wheel from a set that's available. But "most appropriate" does not mean "guaranteed to work" (or even "likely to"!). In particular, it's important to note that they are not for blocking installation on unsupported platforms - if there's no wheel selected, tools will simply use the sdist and install from source, so that just isn't achievable (short of not publishing a sdist, and if you're doing that, you've already thought about the problem enough that you should know to set |
|
Please don't get me wrong, I think Pep425 and the work Daniel did is top notch, a great deal of progress made for the better. What I am trying to politely point out is some of its implementation could be improved. I like to take the view of the simple should be easy and the complex should be possible. In my view pep425 makes a great deal of progress on making the complex simpler, but the simplest of use cases caught me out. I have fully understood the explanation given here and the primary purpose of the tagging system. The abstract from the PEP itself suggests to the contrary:
Yet a py3 defaulted tag indicates to the contrary, most of the time... Not trying to be a pedant, far from it. I am simply trying to articulate my experience and the hurdle I have encountered. As an experienced developer and packager, I am fortunate enough to be able appreciate the nuances involved and in the mean time I will have to manually maintain a python-tag in my builds. Just suggesting this should not be the case or if it is the expected toil to manually maintain this field, it should be mandatory and not default a value which is likely to cause further problems down the road. |
I take mild exception to the fact that you're calling my explanations an "excuse". But I imagine that's not what you intended, so I'll take the view that your meaning just came across badly. I get your point that you expected something from tagged wheels that wasn't actually in line with what they deliver. But you claim things can be "improved", and yet don't give any practical suggestions on how. I don't consider source-code analysis at build time to be a practical option (feel free to create an implementation to prove me wrong...) I also don't consider overly-restrictive tagging to be a good idea - it's way too easy to end up being too restrictive. And I've yet to hear any good argument as to why you don't just specify I think this discussion is just going round in circles now. There's nothing actionable coming out of it, the OP's question has been answered and the issue is already closed. So I'm going to bow out of the conversation at this point. |
Of course I didn't mean to cause any offence. There is absolutely no emotional loading behind any of my discussion here. Quite the opposite, I appreciate your time in responding to me.
I would have done had I known all I do now. I suspect many less informed developers will be getting bitten by this, passing on the problem to the consumers of their builds.
I would hope an individual of your caliber can appreciate, slick tooling should not have to lean on tutorials to get the right outcome, especially in the simplest of examples.
Why recommend something always gets added to achieve the best possible outcome. Make it mandatory in the tooling for a wheel build. It doesn't even warn that a decision has been made on your behalf which could have detrimental repercussions |
|
If Python-Requires metadata is missing, a more specific wheel tag isn't going to save you, as tools running on older versions are going to download the sdist instead. Edit: it's tools that will fall back to the sdist, rather than people explicitly choosing to do it. |
|
Hi Nick, thanks for your feedback. Have already discussed and agreed that this discussion is moot in the presence of a sdist. However that assumes an sdist exists. Only a bdist_wheel in isolation suffers the problems highlighted. |
|
If folks are following the PyPA tutorial, their wheels won't exist in isolation - they'll be creating both a wheel & an sdist, as we actively discourage wheel-only publication. (Private build pipelines are a different story, but if someone's problems are complex enough to require a private build pipeline, then getting the Requires-Python metadata set correctly is likely to be the least of their problems) |
The docs are very good, there is no doubt about that; Some of the best open source documentation around, good stuff. However as previously raised documentation shouldn't exist to cater for inadequate default behaviour. For those that use wheel builds in isolation, the default behaviour should be safe, despite that individuals inexperience or lack of knowledge. It's bitten me, but fortunately I have been able to dig further and realise my mistake. It's probably bitten untold number of others too, who haven't been fortunate enough to realise their mistake, or have but haven't bothered to comment about their experiences and just accepted the state of affairs as it stands. |
|
You're not understanding: distributing wheels without sdists is not an officially supported way of doing things (even though it can be done successfully if certain constraints are met). Therefore, all metadata required for correct installation must be in the sdist. The wheel filename tagging is an efficiency improvement that only applies after the distribution itself has been determined to be potentially acceptable. The way this conversation has gone from my perspective: Now, you may wish to attempt to make your case to the developers of the package building tool you use (i.e. setuptools) that their default for Python-Requires might best be set to something other than the empty string, or at least that the build process should emit a warning if the setting is not specified. But there's no way that the wheel building machinery is ever going to start second-guessing the metadata it receives from the sdist. |
|
A note if you do decide to raise the question of the default Python-Requires string: the status quo is designed such that building for earlier Python releases on a later Python release "just works". This means that most projects do exactly that, and at most test the built artifacts on older Python versions, rather than using the older Python versions to build them. So if the default were to change, it likely wouldn't be to "the version of Python running the build", but rather something like "the oldest version of CPython still receiving security updates when the version of setuptools running the build was released". |
|
Nick, reasonable summary of my position. Appreciate your time in responding so thoroughly. Thanks to all on the thread. |
In jaraco/zipp#37 and pypa/wheel#336, I've learned that downstream consumers seem to care a lot about the value of the python-tag on wheels (and other "built distributions") and that the minimum supported minor version of every supported major version should be included in the python tags.
Then, in jaraco/zipp#42, I've learned that at least one installer workflow has a different interpretation of these tags, suggesting that all supported python versions should be indicated in the tags. I believe this approach is unsustainable, as it requires a lot of twiddling for new Python versions and creates incompatible versions by default.
@dholth As author of PEP 425, can you clarify what are the intentions and limitations of these tags for declaring and selecting the suitable versions for a given environment and what you would recommend for packagers and installers?
Also, it may be worth discussing either in this ticket or a separate one the impending release of Python 4 or Python 3.10, which may have implications for these tags.
cc: @ncoghlan
The text was updated successfully, but these errors were encountered: