Extend twine check to fail on all reasons for which pypi.org would reject a package

[mauvilsa]

Is there an existing issue for this?

• I have searched the existing issues (open and closed), and could not find an existing issue

What keywords did you use to search existing issues?

twine check

Please describe the problem you are attempting to solve with this request

Use twine check to know early enough (pre-commit hook) if an upload of a package to https://pypi.org/ will succeed or fail.

How do you think we should solve this?

twine check should ideally fail for all possible reasons that https://pypi.org/ would reject a package. Or at least the ones that can be determined from the package alone.

Anything else you'd like to mention?

In the past I had a problem uploading a package to pypi, after which I realized that twine check would have already told me that the package was going to be rejected. So I added twine check in a pre-commit hook to avoid this happening again. Unfortunately, it happened again, but this time the package passed with twine check.

Today looking at the docs I see the description of twine check being "Checks whether your distribution’s long description will render correctly on PyPI". On one hand, not great that when I run twine check --help there is no such description. On the other hand, it seems quite misleading that check is only for "long description". I don't know if the original idea was to extend check more and more, and it was never done. Or it has been done and the docs don't say it.

The reason why my package was rejected is:

ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Can't have direct dependency: jsonnet@                                 
         https://github.com/google/jsonnet/zipball/master ; python_version ==   
         "3.13" and extra == "jsonnet". See                                     
         https://packaging.python.org/specifications/core-metadata for more     
         information.

For reference, this relates to google/jsonnet#1172.

[woodruffw]

See also #976, #854, #430 for previous discussions of twine check's limitations.

Long term, I would love to see twine check catch more uploading problems. However, per above, adding more checks is not always trivial: PyPI itself and the packaging standards themselves are moving targets, and there are stateful aspects of PyPI's upload admission process that would be nontrivial for twine check to copy or emulate.

[mauvilsa]

@woodruffw thanks for the info!

From the linked issues it is clear to me that many people want to verify if an upload will succeed before actually uploading. If twine check will never be able to catch all reasons for upload fails, I wonder why clarifications or simpler alternatives haven't been implemented. For example, I could suggest:

• Change twine check --help and the twine docs so that it says what check does now, and warn that it is not intended to catch upload failures.
• Extend the API in warehouse to allow validating a package without uploading.
• With new API, implement in twine CLI a way to validate against the configured pypi. Could be for example in check, or new subcommand, or upload --dry-run, etc.
• Extend github pypa/gh-action-pypi-publish to allow validation without upload.

[woodruffw]

No problem @mauvilsa! These are good (albeit nontrivial) suggestions; I've written some thoughts about each below.

• Change twine check --help and the twine docs so that it says what check does now, and warn that it is not intended to catch upload failures.

I would happily review a change that clarifies the behavior of twine check -- I think the other maintainers will likely have opinions as well, but IMO a clear description of what it does (currently) do vs. not would be an unambiguous boon.

• Extend the API in warehouse to allow validating a package without uploading.

You could open an issue on Warehouse to discuss this, but there are a handful of nontrivial dimensions to this: do "dry-run" uploads require the same auth as normal uploads? How does Warehouse notify people when a dry-run fails (not just a generic error code, but a structured, detailed message)? What guarantees do users have that their packages are not retained during the dry-run? And so forth -- I think these are all surmountable, but they need to be considered in sum.

• With new API, implement in twine CLI a way to validate against the configured pypi. Could be for example in check, or new subcommand, or upload --dry-run, etc.

This points to one of the challenges: PyPI is a single implementation of a handful of standards and informal standards. The upload API is one of those informal standards. If you want "dry running" to be a formal standard that works with uploads other than to PyPI, it needs to go through a PEP process that yields a spec that other upload-receiving indices (e.g. the various stacks provided by AWS, etc.) can feasibly implement. This in turn entails standardizing the upload endpoint itself. Again not insurmountable, but not trivial either!

• Extend github pypa/gh-action-pypi-publish to allow validation without upload.

This is a possibility, although it's worth noting that gh-action-pypi-publish will also have aspects that it can't emulate or guarantee. For example, PyPI will reject uploads if they exceed a particular quota, but uploading clients don't know how close they are to quota ahead of time (they can check it on the web, but it isn't exposed in the APIs).

This may not be an issue, but just as an example of "no dry-run/check API will ever be 100% correct" 🙂

[sigmavirus24]

There's no way for twine to fail for all of the reasons PyPI might fail not least of all because no one can guarantee that between today when we released 6.1 and tomorrow a new validation won't be added to PyPI. To try to constantly keep up with them in both projects is almost needless toil. We'd like to catch more problems for users but we'll never catch them all.