Implement PEP 639

[abravalheri]

Summary of changes

This PR is a "mega PR" that simply make sure all the steps to implement PEP 639 are added at the same time to the main branch.
This is done to avoid problems with incomplete changes:

• We know that tools are being designed and implement PEP 639 compatibility
• We know existing tools know how to understand the current implementation in setuptools
• We don't know how tools would react to something that is not quite a PEP 639 implementation or the exact existing implementation in setuptools.

All the PR are previous reviewed individually.

• Update validate-pyproject to 0.23.0 #4734
• Store license-files in licenses subfolder #4728
• Add initial support for license expression (PEP 639) #4706
• Bump core metadata version to 2.4 #4830
• Prevent deprecated license classifiers from being written to core metadata #4833
• Ensure _apply_pyproject sets field on dist.metadata object not on dist #4834

Good to have according to #4829 (comment):

• The PEP specifies that tools should reject any patterns which contain not-approved chars.

• Partially: Add initial warnings regarding license-files glob patterns. #4838

• Furthermore, it should be an error if a pattern doesn't match any files.

• Partially: Add initial warnings regarding license-files glob patterns. #4838

• Tools should also validate that the license files themselves are utf-8 encoded. Although I don't think that's a priority tbh.

• Currently missing but not blocking

• Lastly, we should probably provide some kind of transition path from tool.setuptools.license-files to project.license-files. Be it as simple as rejecting both together (in which case tool.setuptools.license-files is used) or emitting a warning that tool.setuptools.license-files is deprecated in favor of project.license-files.

• Deprecate tools.setuptools.license-files #4837

Compatibility with other peps:

• Improve interop between Dynamic (METADATA) and dynamic (pyproject.toml) with license/license_expression/license_files #4842

NOTE: I believe that we can simply fast forward the Core Metadata to 2.4 considering that:

• Setuptools emits normalised extra names for Core Metadata:
  https://github.com/pypa/setuptools/blob/v75.8.0/setuptools/_core_metadata.py#L231-L237
  https://github.com/pypa/setuptools/blob/v75.8.0/setuptools/_normalization.py#L100-L107
• Setuptools defers to packaging.requirements.Requirement.__str__ the requirement normalisation, which should also normalise the extra marker¹

https://github.com/pypa/setuptools/blob/v75.8.0/setuptools/_core_metadata.py#L236-L250

Closes

Pull Request Checklist

• Changes have tests
• News fragment added in newsfragments/.
  (See documentation for details)

Footnotes

1. Currently there is a problem with packaging implementation https://github.com/pypa/packaging/issues/845, but for the sake of advancing the setuptools implementation to catch up with the standards we can consider this a "well known bug" that would be solved in the context of pypa/packaging. ↩

[cdce8p]

Opened #4830 with the metadata version bump. I can rebase it after #4706 is merged into the feature branch.

[abravalheri]

Thank you very much @cdce8p!

It seems that a new error with installing tox on cygwin appeared on the CI. I will try to check if it exists in the main branch as it is probably unrelated.

[cdce8p]

One area in which the implementation could still be improved is probably the license-files handling. https://peps.python.org/pep-0639/#add-license-files-key

• The PEP specifies that tools should reject any patterns which contain not-approved chars.
• Furthermore, it should be an error if a pattern doesn't match any files.
• Tools should also validate that the license files themselves are utf-8 encoded. Although I don't think that's a priority tbh.
• Lastly, we should probably provide some kind of transition path from tool.setuptools.license-files to project.license-files. Be it as simple as rejecting both together (in which case tool.setuptools.license-files is used) or emitting a warning that tool.setuptools.license-files is deprecated in favor of project.license-files.

There is also the matter of setup.py and setup.cfg files. My initial intent was to require pyproject.toml for PEP 639. However I haven't tested what happens if one of the setup files overwrites the project metadata.

Side note: Adding support for license expressions to setup.py or setup.cfg is probably a lost cause in itself, as these use a license key already.

--
This looks like a larger list than it is. In particular, they are mostly quality-of-life improvements. So I wouldn't consider them a necessity for the initial implementation.

[abravalheri]

Thank you for the review @cdce8p.

Let's make a checkbox:

• The PEP specifies that tools should reject any patterns which contain not-approved chars.
• Furthermore, it should be an error if a pattern doesn't match any files.
• Tools should also validate that the license files themselves are utf-8 encoded. Although I don't think that's a priority tbh.
• Lastly, we should probably provide some kind of transition path from tool.setuptools.license-files to project.license-files. Be it as simple as rejecting both together (in which case tool.setuptools.license-files is used) or emitting a warning that tool.setuptools.license-files is deprecated in favor of project.license-files.

I would like to tackle at least the last of this points before releasing because we can introduce all the related deprecation warnings/breaking errors at once.

Side note: Adding support for license expressions to setup.py or setup.cfg is probably a lost cause in itself, as these use a license key already.

I agree that there is not much point on updating setup.cfg or setup.py support... although it might be the case if someone sets license_expression = ... there it will just work (remains untested).

[abravalheri]

@cdce8p we are probably in a good state after:

• Add initial warnings regarding license-files glob patterns. #4838
• Add warning for deprecated project.license as TOML table (PEP 639-related) #4840

I am considering if we should ask for volunteers to test in their projects before the release.

---

Possibly License-Expression and License-Files should be reviewed in the context of PEP 643.
We could modify https://github.com/pypa/setuptools/blob/v75.8.0/setuptools/_core_metadata.py#L304-L305. We might also have to mark with _static.Str and _static.List other places, and test it more thoroughly. PEP 639 does allow setuptools to populate License-Files automatically without marking it as dynamic, but the same is not true for setup.py.

---

I am not intending to implement for now:

Tools should also validate that the license files themselves are utf-8 encoded. Although I don't think that's a priority tbh.

But that can be a follow-up PR (not sure where is the best place to implement, maybe https://github.com/pypa/setuptools/blob/v75.8.0/setuptools/command/egg_info.py#L598-L602? maybe in the sdist/bdist_wheel command once we are actually writing the file?)

[abravalheri]

This week we had some friction with a different feature that was "queued for release", and next week I will not be around to help with the release/fixing reports from users. So the earliest we could do is in the week after the next one.

[cdce8p]

next week I will not be around to help with the release/fixing reports from users. So the earliest we could do is in the week after the next one.

I could help out if I see something. Tbh I think the risk here is fairly well contained. As long as we don't break existing configs which this branch shouldn't do, we're likely safe. Anything else which doesn't work 100% could be fixed afterwards as well, once there is more time.