-
Notifications
You must be signed in to change notification settings - Fork 94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore/gen_pypi_cache overhaul #225
chore/gen_pypi_cache overhaul #225
Conversation
Initially I didn't realize that the whole error info was being accumulated for serialization into retval
22bc862
to
1816364
Compare
1816364
to
81bb490
Compare
37aeb7d
to
a36ac96
Compare
Conflation of these reduces clarity of the domain of responsibility
|
||
func pipInstall(pkg string, root string, timeout time.Duration) error { | ||
// Run pip to install just the package, so we can statically analyze it | ||
cmd := exec.Command("pip", "install", "--no-deps", "--target", root, pkg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice! no deps makes this fast.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added unconditional retry here to try and catch transient failures, it added a few minutes more to execution time but not much more
Exciting! I am testing this... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
Why
Statically analyzing which modules are provided by a given package can give us some serious speed improvements.
Pseudoscientific performance graph of execution time before/after this PR:
Additionally, looking at some user reports, relying on
pkgutil
seems to miss packages that are valid. As an example:$ mkdir foo/bar/baz $ touch foo/bar/baz/blix.py $ python -c 'import foo.bar.baz.blix'
exits cleanly, though
pkgutil
would fail to see insidefoo
due tofoo
not being a module itself.A real-world example of no longer overreporting modules:
What changed
test-one
subcommand for easy manual testingpip install --no-deps
to limit search blast radius. Important for "extension"-style modules which depend on the underlying package, which would obscure which package supplies the extension (flaskext.mysql
, as an example, may also falsely reportflask
as a provided module)Test plan
$ go run ./gen_pypi_map test -force
observe that it is faster
observe output