Rye: What's Happening!?

[mitsuhiko]

The last few weeks have been quite quiet here. That has a few different reasons. On the one I underwent a surgery and afterwards I took a vacation. Now with the excuses out of the way, there is another reason why there are not many updates. I evaluated various different ways to get rid of pip-tools and I just slowly started to realize how much complexity there is in these underlying tools and how brittle the interactions with the build tools are. That does not discourage me, but that showed me that I need to take a step back and analyze better what is needed to make this a successful package manager.

My immediate next steps are to better document the issues I'm running into ripping out pip-tools to have folks chip in that might know more than me about this particular issue.

Separately to that another limiting factor at the moment is the total lack of tests in rye. This is something that really impedes any progress at the moment and I really need to start solving. This is most likely what's going to happen within the next few months. I just need to create a bunch of supporting infrastructure for those tests not to take ages. I already started with this a few times now, and every time I did not dare to put them up anywhere because of how long the few tests I created take to run.

With that, expect more updates in the next few weeks.

[vinitkumar]

Thanks for the update.

My immediate next steps are to better document the issues I'm running into ripping out pip-tools to have folks chip in that might know more than me about this particular issue.

By ripping off pip-tools, do you mean that you will implement what it does in rye itself, written in Rust or something fundamentally different?

[mitsuhiko]

pip-tools has various limitations that cause problems. The biggest issue is that it's not capable of producing portable lock files, the second issue is that it cannot perform the resolution/locking across virtualenvs. Rye needs to maintain a virtualenv in addition for interpreter just to have pip-tools work. The combination of both of those things, bundled with the particular implementation that pip-tools has, results in very slow operations. Slow enough that the experience really suffers. It also results in a user experience that's far from what I think would be a good one.

I'm not yet sure what the solution looks like, and I don't want to speculate yet. What I believe is that whatever the solution is, it needs to work so that also non rye can use it as the fundamentals would need to be building blocks. pdm already does quite a bit of this today and one of the experiments I had just used pdm internals under the hood. However even pdm is not able to install all the test packages I tried with and it also some internals are not stable, so just using that doesn't really work either.

[vinitkumar]

Alright, thanks for the clear explanation.

Withpip-tools, we can generate hashes for the dependencies. pip-compile --generate-hashes. Is that not enough for this use-case and are they not portable enough?

[cnpryer]

Hacktoberfest goals! :)

[LeroyK111]

You're doing really well.