Tracking issue for cleaning up python lib formulae

[cho-m]

A dedicated tracker for any tasks to handle based on decision in #157500.

For Homebrew/core:

• Triage python library formulae:
  • Remove python library formulae that do not meet our acceptable requirements and have low number of on-request installs
  • Deprecate python library formulae that do not meet our acceptable requirements and have high number of on-request installs (or have been around for a while in Homebrew/core). We can do a 3mo deprecation + 3mo disable period.
  • Convert/revert Python application formulae that are installed to prefix to instead use virtualenvs
  • Document anything remaining as approved exceptions

Extras for Homebrew/brew:

• Improve audit checks, particularly for --new formulae, e.g. audit: check for Python-wide site-package usage brew#16663
• Improve brew doctor and --ignore-dependencies "recommendation" for removed formulae to avoid users blindly running commands
• Optional: Maybe perform automatic "migration" to new rebuild/revision for removed formulae (which would also need to handle tab of recursive dependents)

---

Quick table based on 90-day install-on-request analytics data for formulae that may expose libs in HOMEBREW_PREFIX/lib. Also added whether the formula provides any files in bin and non-pure python libs.

Formula	90-day installs	90-day rank	# direct usage	Comment	bin/* cmd?	Ext libs
pycparser	38,795	136	4	dependency of cryptography	❎	❎
numpy	38,584	138	44	keep, Homebrew/brew#16662	✅︎	✅︎
cffi	33,997	154	7	dependency of cryptography	❎	✅︎
mercurial	16,247	273	4	keep	✅︎	✅︎
pillow	14,684	293	21	keep	❎	✅︎
pyqt@5	10,765	352	5	keep	✅︎	✅︎
pyqt	10,336	363	3	keep	✅︎	✅︎
python-setuptools	7,381	463	87		❎	❎
sip	6,932	477	2		✅︎	❎
cryptography	6,285	511	69	keep, Homebrew/brew#16662	❎	✅︎
scipy	6,248	512	4	keep, Homebrew/brew#16662	❎	✅︎
six	2,887	806	4		❎	❎
python-matplotlib	2,823	820	0		❎	✅︎
python-tabulate	1,771	1083	0	keep #175440	✅︎	❎
certifi	1,707	1109	164	keep, Homebrew/brew#16662	❎	❎
python-lxml	1,648	1135	0	deprecate, #171823	❎	✅︎
python-packaging	1,547	1189	1	deprecate #176511	❎	❎
pyyaml	1,319	1293	3	deprecate #176591	❎	✅︎
python-markdown	1,293	1313	0	keep #175442, decide on rename	✅︎	❎
python-build	1,564	1176	1	reverted to venv, #165713 Decide if rename?	✅︎	❎
pymol	964	1577	0		✅︎	✅︎
pygit2	904	1630	0		❎	✅︎
pymupdf	824	1717	1		✅︎	✅︎
python-argcomplete	813	1731	0		✅︎	❎
lit	678	1908	1		✅︎	❎
wxpython	639	1962	0		✅︎	✅︎
python-ply	559	2101	0	deprecate, #171825	❎	❎
python-chardet	494	2260	0	keep #175446	✅︎	❎
pyqt-builder	214	3530	3		✅︎	❎
ly	58	6025	0	keep bin, decide on lib.	✅︎	❎

Table for completed tasks (i.e. merged PRs), excluding removals:

Formula	90-day installs	90-day rank	# direct usage	Decision/PR	bin/* cmd?	Ext libs
docutils	32,666	162	10	reverted to venv, #168183	✅︎	❎
python-cryptography	24,100	205	0	renamed to cryptography	n/a	n/a
python-requests	4,118	650	0	deprecated, #166056	❎	❎
python-certifi	2,262	928	0	renamed to certifi	n/a	n/a
python-pytz	1,104	1447	0	deprecated, #168071	❎	❎
python-typing-extensions	1,033	1499	1	deprecated, #168114	❎	❎
python-urllib3	877	1667	1	deprecated, #166056	❎	❎
asciidoc	696	1886	25	reverted to venv, #165317	✅︎	❎
python-dateutil	644	1959	0	deprecated, #168071	❎	❎
python-psutil	603	2019	0	deprecated, #168071	❎	✅︎
python-trove-classifiers	507	2224	0	deprecated, #168181	❎	❎
python-idna	441	2385	1	deprecated, #166056	❎	❎
python-charset-normalizer	218	3503	1	deprecated, #166056	❎	❎

Collapsed table for removed formulae

Formula	90-day installs	90-day rank	# direct usage	Decision/PR	bin/* cmd?	Ext libs
python-markupsafe	1,742	1093	0	removed, #165568	n/a	n/a
python-platformdirs	447	2369	0	removed, #163603	n/a	n/a
python-boto3	398	2511	0	removed, #163229	n/a	n/a
python-pyparsing	389	2541	0	removed, #168201	❎	❎
python-jinja	366	2631	0	removed, #165495	n/a	n/a
python-hatchling	339	2749	0	removed, #166358	n/a	n/a
python-distlib	247	3277	0	removed, #163570
meson-python	235	3363	0	removed, #166060	❎	❎
python-setuptools-scm	179	3879	0	removed, #168073	❎	❎
python-flit-core	174	3942	0	removed, #168201	❎	❎
python-click	171	3983	0	removed, #165496
python-toml	170	4000	0	removed, #165493
python-sympy	169	4010	0	removed, #165494
python-mako	169	4016	0	removed, #165568
python-pluggy	149	4239	0	removed, #166358
python-pycurl	148	4247	0	removed, #164368
python-mutagen	116	4715	0	removed, #164365
python-pathspec	114	4742	0	removed, #166358
python-brotli	107	4869	0	removed, #165112
python-attrs	102	4937	0	removed, #163625
python-networkx	95	5069	0	removed, #166185
python-botocore	84	5306	0	removed, #163229
python-magic	81	5374	0	removed, #164158
python-filelock	78	5446	0	removed, #164940
python-openapi3	71	5638	0	removed, #163442
python-cli-helpers	70	5666	0	removed, #163436
python-colorama	69	5694	0	removed, #163437
python-distro	67	5750	0	removed, #164160
python-cycler	64	5839	0	removed, #166060	❎	❎
python-abseil	62	5908	0	removed, #163343
python-regex	61	5918	0	removed, #164146
python-docopt	55	6100	0	removed, #164149
python-pbr	55	6134	0	removed, #164154
python-configargparse	52	6242	0	removed, #163438
python-hatch-vcs	52	6243	0	removed, #164144
python-json5	52	6244	0	removed, #163642
python-xlsxwriter	51	6288	0	removed, #163646
python-kiwisolver	49	6382	0	removed, #166060	❎	✅︎
python-requests-oauthlib	48	6421	0	removed, #163590
python-hatch-fancy-pypi-readme	47	6462	0	removed, #164085
python-prompt-toolkit	47	6463	0	removed, #163576
python-termcolor	46	6511	0	removed, #163378
python-rich	45	6557	0	removed, #163623
python-asn1crypto	43	6655	0	removed, #163363
python-oauthlib	43	6656	0	removed, #163712
python-cachetools	41	6745	0	removed, #163425
python-pyproject-hooks	39	6835	0	removed, #166183
python-websocket-client	39	6836	0	removed, #163645
python-dicttoxml	32	7219	0	removed, #163567
python-mpmath	32	7220	0	removed, #165494
python-markdown-it-py	30	7362	0	removed, #164086
python-anytree	29	7431	0	removed, #163424
python-wcwidth	28	7489	0	removed, #163592
python-msgpack	27	7555	0	removed, #164148
python-jmespath	13	9027	0	removed, #163229
python-configobj	7	10540	0	removed, #163565
python-mdurl	7	10541	0	removed, #164086
python-s3transfer	6	10927	0	removed, #163229

[carlocab]

Should probably keep pygit2 -- it links with libgit2, and vendoring it into its dependents leads to really long build times (due to revision bumps) when libgit2 is updated.

[LecrisUT]

I've opened an issue to point out a potential problems with the build process, specifically affecting python projects which have setuptools-scm as a dependency

[offizium-berndstorath]

What is the reason for removing python-requests?

[SMillerDev]

What is the reason for removing python-requests?

It's a library, not something that's useful on its own. People are better off adding it to their requirements.txt if they need it somewhere.

[saurik]

FWIW, my pitch for python-packaging -- apparently now deprecated and soon to be removed due to #176511 -- is that I see it being used as part of peoples' meson builds, and the workaround for it not being part of the distribution is both annoying and non-obvious (I certainly wouldn't have done it correctly), and yet winds up being forced on everyone attempting to compile such packages, whether as a developer automating their build system or as a user who wants to one-off build a random project they checked out which uses meson.

You can actually see this at work in the glib formula: instead of merely marking python-packaging as a build dependency, it instead must vendor a specific version of the package -- which already feels wrong, as this is something that is going to have to get updated, and yet is going to be more coupled to python than glib -- and then futz around with the PYTHON_PATH to inject it into the system python search path, as, apparently, doing this work using a virtual env and passing that separate python context to meson causes glib to not work correctly, per the giant comment in the formula.

homebrew-core/Formula/g/glib.rb

Lines 55 to 58 in 302c6d8

	resource "packaging" do
	url "https://files.pythonhosted.org/packages/51/65/50db4dda066951078f0a96cf12f4b9ada6e4b811516bf0262c0f4f7064d4/packaging-24.1.tar.gz"
	sha256 "026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002"
	end

homebrew-core/Formula/g/glib.rb

Lines 71 to 81 in 302c6d8

	# We don't use a venv as virtualenv_create runs `ENV.refurbish_args`. This causes `gint64`
	# to be detected as `long` rather than `long long` on macOS which mismatches with `int64_t`.
	# Although documented as valid (https://docs.gtk.org/glib/types.html#gint64), it can cause
	# ABI breakage if type changes between releases (e.g. seen in `[email protected]`) and it breaks
	# assumptions made by some dependents. Also, GNOME prefers equivalence of types but cannot
	# require it due to ABI impact - https://gitlab.gnome.org/GNOME/glib/-/merge_requests/2841
	resource("packaging").stage do
	system "python3.12", "-m", "pip", "install", "--target", share/"glib-2.0",
	*std_pip_args(prefix: false, build_isolation: true), "."
	end
	ENV.prepend_path "PYTHONPATH", share/"glib-2.0"

I also found a mention by a maintainer of meson (@eli-schwartz) in a discussion of what modules might be useful to include in a standalone copy of meson into which one cannot install additional dependencies--which, FWIW, seems to be the case with Homebrew now, given that python3 is externally managed and these packages that are normally used to accomplish the same goals are largely being deprecated--and packaging is specifically and narrowly called out as "probably a good thing to include" as it is apparently being commonly in various meson build scripts as a replacement for some useful functions from distutils.

mesonbuild/meson#13007 (reply in thread)

The official MSI installer for meson on Windows doesn't include any third-party modules so that probably encourages users to avoid overreliance on such modules. However the packaging module does contain the version parsing and comparison code that replaces distutils functions, so that's probably a good thing to include just in case as people may have ported away from distutils in their script commands.

@cho-m @woodruffw Is there any hope of getting a stay of execution on the python-packaging formula?

[carlocab]

Is there any hope of getting a stay of execution on the python-packaging formula?

The discussion above seems like good reasoning for this for me, but CC @Homebrew/core for thoughts.

[MikeMcQuaid]

You can actually see this at work in the glib formula: instead of merely marking python-packaging as a build dependency, it instead must vendor a specific version of the package -- which already feels wrong, as this is something that is going to have to get updated, and yet is going to be more coupled to python than glib -- and then futz around with the PYTHON_PATH to inject it into the system python search path, as, apparently, doing this work using a virtual env and passing that separate python context to meson causes glib to not work correctly, per the giant comment in the formula.

This seems like a very good reasoning for me along to have python-packaging as a formula.

Really, anything we're repeatedly resourceing in multiple formulae in this sort of non-trivial way feels better handled as a resource.

[SMillerDev]

Really, anything we're repeatedly resourceing in multiple formulae in this sort of non-trivial way feels better handled as a resource.

You mean resources where we have to mess with pythonpath might be eligible to be formulae right? Not just "anything that we resource a lot" because that used to be the strategy.

[MikeMcQuaid]

You mean resources where we have to mess with pythonpath might be eligible to be formulae right? Not just "anything that we resource a lot" because that used to be the strategy.

Resources where we're having to write elaborate explanations and provide workarounds like this and we have helpful community members like @saurik taking the time to explain why.

I don't think a "follow this rule and you don't have to think about it" approach is going to work in either direction (not that I'm claiming you're saying that either, to be clear).

[carlocab]

Un-disable!d python-packaging at #183426.