Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prefer the 'apt_pkg' module over 'apt' (SC-1516) #2744

Merged
merged 11 commits into from
Oct 2, 2023
Merged

Conversation

renanrodrigo
Copy link
Member

@renanrodrigo renanrodrigo commented Sep 21, 2023

Why is this needed?

This PR solves all of our problems because we are trying to standardize the use of python3-apt through the codebase. After some discussions with @julian-klode in the past, we have decided to stick to apt_pkg instead of apt when possible, so we would:

  • improve performance
  • rely on a single mechanism to mess with apt data
  • guarantee that no hidden functionality in apt is breaking our isolation of configuration, given that we are responsible for the esm-apt instance
  • open possibilities to implement more features (as reporting progress, for instance) using apt_pkg capabilities.

Test Steps

This is just a medium-size refactor; no new features introduced, so CI should pass without major changes (although adjustments in unit tests were of course needed because we still test some implementation details and isolation is not 100%)

It is possible to verify performance gains, maybe where it is more impactful: run pro security-status on an updated unattached bionic (any release, really) machine both with and without this patch and compare the differences.

Checklist

  • I have updated or added any unit tests accordingly
  • I have updated or added any integration tests accordingly
  • Changes here need to be documented, and this was done in:

Does this PR require extra reviews?

  • Yes - it would be nice to have @julian-klode 's eyes on it before merging
  • No

@github-actions
Copy link

github-actions bot commented Sep 21, 2023

Jira: SC-1516

GitHub Issues: No GitHub issues are fixed by this PR. (No commits have Fixes: #### references)

Launchpad Bugs: No Launchpad bugs are fixed by this PR. (No commits have LP: #### references)

Documentation: The changes in this PR do not require documentation changes.

👍 this comment to confirm that this is correct.

@renanrodrigo renanrodrigo force-pushed the prefer_apt_pkg branch 4 times, most recently from c77b2b7 to a6448c9 Compare September 21, 2023 19:11
@renanrodrigo renanrodrigo changed the title WIP: Prefer the 'apt_pkg' module over 'apt' Prefer the 'apt_pkg' module over 'apt' (SC-1516) Sep 22, 2023
@renanrodrigo renanrodrigo marked this pull request as ready for review September 22, 2023 16:59
uaclient/apt.py Outdated Show resolved Hide resolved
uaclient/apt.py Show resolved Hide resolved
@renanrodrigo
Copy link
Member Author

renanrodrigo commented Sep 22, 2023

The curious thing is that although benchmarking with timeit have shown the apt_pkg loops to be way faster (reducing some calls from almost 3s to around 0.4s), when running the commands the performance is actually worse. Worth taking a second look. sorted out, was a stupid implementation detail, now fixed

@renanrodrigo renanrodrigo force-pushed the prefer_apt_pkg branch 5 times, most recently from 25ab041 to e538dc7 Compare September 25, 2023 22:07
@renanrodrigo
Copy link
Member Author

CI Failures unrelated - a cloud timeout and #2756

uaclient/apt.py Outdated Show resolved Hide resolved
uaclient/apt.py Outdated Show resolved Hide resolved
uaclient/apt.py Show resolved Hide resolved
uaclient/apt.py Outdated Show resolved Hide resolved
uaclient/apt.py Outdated Show resolved Hide resolved
@julian-klode
Copy link
Contributor

I'll try to have a look tomorrow

Copy link
Collaborator

@orndorffgrant orndorffgrant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great refactor +1
Just one small nit

uaclient/version.py Show resolved Hide resolved
Remove the global mock and install python-apt from source. This way we
can have access to the 'apt' and 'apt_pkg' modules during test
execution.
This revealed a fixture we missed on the apt update functionality and
some mypy complains we could have addressed, and simplified a couple
tests.

Signed-off-by: Renan Rodrigo <[email protected]>
As we are not creating the cache using the apt module anymore, a new
helper function is needed to ensure the structure is there before we set
up the sources.list files

Signed-off-by: Renan Rodrigo <[email protected]>
Also fix a unit test that was passing based on luck

Signed-off-by: Renan Rodrigo <[email protected]>
It is more resilient now, but we can log success and failure using
the AcquireProgress object

Signed-off-by: Renan Rodrigo <[email protected]>
Instead of performing the check at the module level, we decorate the
functions that use the module without instantiating a Cache object (in
the Cache case, we initialize apt_pkg separately)

Signed-off-by: Renan Rodrigo <[email protected]>
@renanrodrigo
Copy link
Member Author

@julian-klode we need to merge for the release, but any further review or comment is welcome - we can address those in next iteration

@renanrodrigo renanrodrigo merged commit bf7671b into main Oct 2, 2023
22 of 23 checks passed
@renanrodrigo renanrodrigo deleted the prefer_apt_pkg branch October 2, 2023 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants