Upgrade tooling on the project. #1329

almet · 2024-09-29T21:54:31Z

Replace black by ruff, as it's quicker ;
Drop python 3.7, since it's end of life since last year ;
Use uv wherever possible as a replacement for pip, as it's way faster to run, add an uv.lock file which will be synced before the releases and published here ;
Remove tox, it's too complex for this project and can easily be replaced by uv ;
Apply ruff formatting ;
Update the makefile accordingly ;
Update the CI accordingly

I'm not sure why, but pytest-libfaketime is currently not working when running inside uv, at least on an apple silicon machine. I uninstalled it otherwise the test couldn't run, failing with the following error:

uv run --extra dev pytest .
re-exec with libfaketime dependencies
dyld[16935]: terminating because inserted dylib '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' could not be loaded: tried: '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file)
dyld[16935]: tried: '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file)

- Replace black by ruff, as it's quicker ; - Use `uv` wherever possible as a replacement for pip, as it's way faster to run, add an `uv.lock` file which will be synced before the releases and published here ; - Remove tox, it's too complex for this project and can easily be replaced by `uv` ; - Apply `ruff` formatting ; - Update the makefile accordingly ; - Update the CI accordingly

almet · 2024-09-29T22:25:22Z

Tests are failing with python 3.10 + sqlite, but I cannot reproduce locally with the following command:

TESTING_SQLALCHEMY_DATABASE_URI="sqlite:///budget.db"; uv run --python 3.10 --extra dev --extra database pytest .

Edit: It's fixed after a re-run.

almet · 2024-09-29T23:01:49Z

Curious about your feelings on this @zorun ?

Glandos

I'm still alive, and sometimes, I even have time to review!

pyproject.toml

Co-authored-by: Glandos <[email protected]>

zorun · 2024-10-01T20:58:12Z

Keep in mind that I tend to be conservative for this kind of reviews ;) I once inherited a project using poetry and I really didn't like it.

Here are general questions:

What would be the main advantage and disadvantages of this change? (I don't buy the argument "it's newer fancy tooling"). Do we have performance issues with the current tooling?
What guarantee do we have that the new tooling will still work reliably X years from now?
Does it affect releases? Would the project still be installable via pip? If a user installs the project with pip, do they get the same dependencies that we have in the lockfile? Can we run tests for that situation?
Does it affect users directly or indirectly in other ways?
Does it create a higher bar for entry for new contributors?

almet · 2024-10-01T21:53:40Z

No problem, thanks for the questions.

I believe part of the motivation behind this change is personal: putting back this project as a way to experiment with "new shiny tools". It gives me energy, and as such I believe it will be net-beneficial for the project in the end. More fun means it will get easier to maintain (at least for me, but you can have a different opinion).

With that out of the way, let's discuss the points you bring here 👍

What would be the main advantage and disadvantages of this change?

I believe the main pro is speedup. I personally also like to be able to install different versions of python in a simple way with uv python install in just a few seconds, as it makes it easier to run tests without having to install these manually with another tool.

Do we have performance issues with the current tooling?

I don't believe we have performance issues with the current tooling, no.

What guarantee do we have that the new tooling will still work reliably X years from now?

My feeling is that uv is here to stay, but that's a reasonable question to ask. The postscriptum on the announce covers it a bit:

What does that mean for the future of these tools? Here is my take on this: for the community having someone pour money into it can create some challenges. For the PSF and the core Python project this is something that should be considered. However having seen the code and what uv is doing, even in the worst possible future this is a very forkable and maintainable thing. I believe that even in case Astral shuts down or were to do something incredibly dodgy licensing wise, the community would be better off than before uv existed.

Does it affect releases?

No, not really. It can affect how we release, because the tools we use would be installed by uv.

Would the project still be installable via pip?

Yes.

If a user installs the project with pip, do they get the same dependencies that we have in the lockfile?

When using pip, the dependencies are not compiled by the uv lockfile, but by what's defined in the pyproject.toml, so I believe we're covered here.

Can we run tests for that situation?

I don't think there is a need for it :-)

Does it affect users directly or indirectly in other ways?

It's mainly a DX change.

Does it create a higher bar for entry for new contributors?

I believe it's actually the other way around, as it makes things simpler and faster.

zorun · 2024-10-01T22:20:31Z

I believe part of the motivation behind this change is personal: putting back this project as a way to experiment with "new shiny tools". It gives me energy, and as such I believe it will be net-beneficial for the project in the end. More fun means it will get easier to maintain (at least for me, but you can have a different opinion).

Thanks, I wanted to hear your motivation. If it motivates you, that's good, but let's not break things for free.

Actually, thinking about it, I think my motivation comes from repairing broken things, so maybe we make the perfect combo :D

What would be the main advantage and disadvantages of this change?

I believe the main pro is speedup. I personally also like to be able to install different versions of python in a simple way with uv python install in just a few seconds, as it makes it easier to run tests without having to install these manually with another tool.

This is a good argument.

If a user installs the project with pip, do they get the same dependencies that we have in the lockfile?

When using pip, the dependencies are not compiled by the uv lockfile, but by what's defined in the pyproject.toml, so I believe we're covered here.

Can we run tests for that situation?

I don't think there is a need for it :-)

Let me be more clear, because this is a general grip I also had with poetry.

Lockfiles are very good when working between developers: we are sure we have the exact same version of dependencies. However, users will have different dependencies because they don't use the lockfiles, they use pip.

If we run tests with the dependencies from the lockfile, we don't test what the actual users will install, and this is quite bad.

We could run tests using the dependencies from pyproject.toml instead of the lockfile, but then, what would be the point of the lockfile? In that case, let's just use pyproject.toml also for development.

What do you think? Did I miss something obvious?

almet · 2024-10-01T22:34:46Z

Thanks, I wanted to hear your motivation. If it motivates you, that's good, but let's not break things for free.

Actually, thinking about it, I think my motivation comes from repairing broken things, so maybe we make the perfect combo :D

🙌 🤭

Let me be more clear, because this is a general grip I also had with poetry.

Lockfiles are very good when working between developers: we are sure we have the exact same version of dependencies. However, users will have different dependencies because they don't use the lockfiles, they use pip.

If we run tests with the dependencies from the lockfile, we don't test what the actual users will install, and this is quite bad.

We could run tests using the dependencies from pyproject.toml instead of the lockfile, but then, what would be the point of the lockfile? In that case, let's just use pyproject.toml also for development.

Oh, that's nice, and I understand better now the reason why we have the "minimal" set of dependencies in our tests. I actually think this is a good idea.

I see the .lock file as a "known working set" of dependencies. I don't believe we need to ship it to other users, but that could be something documented, and also part of the release, for those who want to have more control over the environment they are running the project into.

For instance, for the Docker image, we could ensure that it ships with exactly the versions that were present at the time of the build, and I believe that would be a good thing. Not a requirement, but something useful for some people.

But, to be honest, these .lockfiles are just starting to make sense to me.

almet · 2024-10-01T22:36:26Z

Also: thanks for the tone of the discussion, I told you already but it's been a pleasure to disagree with you, and find out how we can better understand each other and come to the same conclusion.

(This is probably because we're not trying to push our decisions onto each other, but really care about what we think, and why)

Glandos · 2024-10-02T13:24:10Z

I believe part of the motivation behind this change is personal: putting back this project as a way to experiment with "new shiny tools". It gives me energy, and as such I believe it will be net-beneficial for the project in the end. More fun means it will get easier to maintain (at least for me, but you can have a different opinion).

It's also my point-of-view. Since I knew nothing about uv, I checked it. It's pretty new. It's pretty cool. The only reassuring stuff is that it's made by the same team doing ruff which is here to stay. Otherwise, uv would have been the "15 competing standard" regarding Python dependency management.

Oh, that's nice, and I understand better now the reason why we have the "minimal" set of dependencies in our tests. I actually think this is a good idea.

I remember taking part in this, and this was a really good move, because it also tests installation in every cases. I also found myself in situation where I needed to install something (hello Ansible) at not the very last version, and was happy it worked :)

almet · 2024-10-28T22:45:52Z

Hey, @zorun, after some rest, what's your feeling on this? Should we merge or close?

zorun

I'm not opposed on principle, but there are some unknowns for me. I have left a few comments.

Here is the main missing stuff, I think:

did you check that all instructions in docs/contributing.md still work after this change?
what's the position on the uvlock? As I said, I'm a bit dubious about it because users will install IHM via pip, so I believe we shouldn't use the uvlock at all. Does uv use the lock by default? Once we reach a consensus, please document the result somewhere in the doc.
should the Dockerfile use uv or continue using pip?

zorun · 2024-10-31T12:31:10Z

.github/workflows/ci.yml


  test:
-    # Dependency on linting to avoid running our expensive matrix test for nothing


Why do you remove this comment and the other ones below?

I'm actually not sure! (tbh I don't remember). Let's add them back.

zorun · 2024-10-31T12:31:33Z

.github/workflows/ci.yml

    - name: Change dependencies to minimal supported versions
-      run: sed -i -e 's/>=/==/g; s/~=.*==\(.*\)/==\1/g; s/~=/==/g;' pyproject.toml
+      run: sed -i -e '/requires-python/!s/>=/==/g; /requires-python/!s/~=.*==\(.*\)/==\1/g; /requires-python/!s/~=/==/g;' pyproject.toml


What does this do? Maybe add a comment.

zorun · 2024-10-31T12:32:01Z

.github/workflows/ci.yml

-      if: matrix.database == 'mariadb'
-      env:
-        TESTING_SQLALCHEMY_DATABASE_URI: 'mysql+pymysql://root:ihatemoney@localhost:3306/ihatemoney_ci'
+        TESTING_SQLALCHEMY_DATABASE_URI: ${{ matrix.database == 'sqlite' && 'sqlite:///budget.db' || matrix.database == 'postgresql' && 'postgresql+psycopg2://postgres:ihatemoney@localhost:5432/ihatemoney_ci' || 'mysql+pymysql://root:ihatemoney@localhost:3306/ihatemoney_ci' }}


I don't find this very readable, what's wrong with the current approach?

I can add a comment there. It's just a different approach, that I find better because it has less repetition. I can surely add a comment here and format it better so it's easier to parse :-)

almet · 2024-10-31T21:29:50Z

did you check that all instructions in docs/contributing.md still work after this change?

No, not yet. I will if we decide to go this way.

what's the position on the uvlock? As I said, I'm a bit dubious about it because users will install IHM via pip, so I believe we shouldn't use the uvlock at all. Does uv use the lock by default? Once we reach a consensus, please document the result somewhere in the doc.

I believe uv.lock files will allow us to ship a known working set of dependencies, that can be used for instance in the Docker file.

But, it won't be used for installations via pip. I think we should keep it, but it won't have any effect on the distribution via PyPI.

almet added 8 commits September 29, 2024 23:47

FIXUP: update CI

22198e8

FIXUP: remove check-doc from the CI, it's part of the main workflow

0f8011a

FIXUP: install python via uv

ef246bb

FIXUP: Add pytest-libfaketime to the deps

fd22912

FIXUP: Add extra deps database when running the tests

21ba777

FIXUP: Change the name of the CI on GHA

3a17200

FIXUP: Do not sed-replace requires-python

895e6f6

almet requested review from Glandos and zorun and removed request for Glandos September 29, 2024 23:01

Glandos reviewed Sep 30, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

almet and others added 2 commits September 30, 2024 14:24

Update pyproject.toml

a65be37

Co-authored-by: Glandos <[email protected]>

Update pyproject.toml

4e2090b

Co-authored-by: Glandos <[email protected]>

Glandos approved these changes Sep 30, 2024

View reviewed changes

zorun requested changes Oct 31, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade tooling on the project. #1329

Upgrade tooling on the project. #1329

almet commented Sep 29, 2024 •

edited

Loading

almet commented Sep 29, 2024 •

edited

Loading

almet commented Sep 29, 2024

Glandos left a comment

zorun commented Oct 1, 2024

almet commented Oct 1, 2024

zorun commented Oct 1, 2024 •

edited

Loading

almet commented Oct 1, 2024

almet commented Oct 1, 2024

Glandos commented Oct 2, 2024

almet commented Oct 28, 2024

zorun left a comment

zorun Oct 31, 2024

almet Oct 31, 2024

zorun Oct 31, 2024

zorun Oct 31, 2024

almet Oct 31, 2024

almet commented Oct 31, 2024


		test:
		# Dependency on linting to avoid running our expensive matrix test for nothing

Upgrade tooling on the project. #1329

Are you sure you want to change the base?

Upgrade tooling on the project. #1329

Conversation

almet commented Sep 29, 2024 • edited Loading

almet commented Sep 29, 2024 • edited Loading

almet commented Sep 29, 2024

Glandos left a comment

Choose a reason for hiding this comment

zorun commented Oct 1, 2024

almet commented Oct 1, 2024

zorun commented Oct 1, 2024 • edited Loading

almet commented Oct 1, 2024

almet commented Oct 1, 2024

Glandos commented Oct 2, 2024

almet commented Oct 28, 2024

zorun left a comment

Choose a reason for hiding this comment

zorun Oct 31, 2024

Choose a reason for hiding this comment

almet Oct 31, 2024

Choose a reason for hiding this comment

zorun Oct 31, 2024

Choose a reason for hiding this comment

zorun Oct 31, 2024

Choose a reason for hiding this comment

almet Oct 31, 2024

Choose a reason for hiding this comment

almet commented Oct 31, 2024

almet commented Sep 29, 2024 •

edited

Loading

almet commented Sep 29, 2024 •

edited

Loading

zorun commented Oct 1, 2024 •

edited

Loading