Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests/test_CLI.py: create $GNUPGHOME on the fly #31

Merged
merged 7 commits into from
Jan 16, 2025

Conversation

josch
Copy link

@josch josch commented Jul 16, 2024

  • avoid expiration of keys by re-creating them
  • prevent gnupg version on developer's system being incompatible with $GNUPGHOME in git

Storing binary data is bad because:

  • git is not good at handling binary data
  • binary data is harder to inspect (remember the xz incident)

With this commit, test_clearsign fails with "TypeError: 'NamedFile' object is not iterable" but I read that this issue is addressed by #1 so I'm not adding this here.

@josch
Copy link
Author

josch commented Jul 16, 2024

@jo-so-nx I rebased your commits from #1 on top of mine, except for 385f4d6 as the expiration extension by 50 years is not necessary if the key is generated on-the-fly.

What do you think?

@jo-so-nx
Copy link

jo-so-nx commented Aug 8, 2024

@jo-so-nx I rebased your commits from #1 on top of mine, except for 385f4d6 as the expiration extension by 50 years is not necessary if the key is generated on-the-fly.

What do you think?

I'm fine with this. You can pick and modify them as you like.

I must admit we have abandoned bmaptool in our project and have no longer the infrastructure to test anything.

@josch
Copy link
Author

josch commented Aug 10, 2024

I'm fine with this. You can pick and modify them as you like.

Thank you!

I must admit we have abandoned bmaptool in our project and have no longer the infrastructure to test anything.

I'm a bit worried about the same having happened to Yocto as there was very little activity in this project since March 2024. What is the status @JPEWdev @twoerner?

@zeha
Copy link

zeha commented Nov 9, 2024

ping @JPEWdev @twoerner please, could you check/apply this? thanks :)

@JPEWdev JPEWdev self-requested a review November 19, 2024 16:55
@JPEWdev
Copy link
Collaborator

JPEWdev commented Nov 19, 2024

I'm looking at it, but I've lost the ability to trigger workflows, so until that's done I can't run the CI to test this

@JPEWdev
Copy link
Collaborator

JPEWdev commented Jan 13, 2025

@josch Sorry, I cannot figure out why I can't retrigger the CI on this. Can you please rebase your branch on main and push it again? That should trigger it.

josch and others added 5 commits January 13, 2025 17:29
 - avoid expiration of keys by re-creating them
 - prevent gnupg version being incompatible with $GNUPGHOME in git

Storing binary data is bad because:

 - git is not good at handling binary data
 - binary data is harder to inspect (remember the xz incident)
Passing 0xFFFFFFFFFFFFFFFF to read causes python to complain about:

    OverflowError: cannot fit 'int' into an index-sized integer

Signed-off-by: Jörg Sommer <[email protected]>
The current tests do not take into account whether the `gpg` package has
been installed or not. If it is missing, the tests should be skipped.

Furthermore, the output of the tests must be checked in order to decide
whether tests fail due to an exception or whether the desired error message
is displayed.

Signed-off-by: Jörg Sommer <[email protected]>
The verification of PGP signatures had some flaws and didn't work, because
the Python API and the GPG interface have changed. Inline signatures were
not detected, because of a comparison of string and byte array. And even
after this the code failed, because `sig.status` is no longer available.

Signed-off-by: Jörg Sommer <[email protected]>
@josch
Copy link
Author

josch commented Jan 13, 2025

Can you please rebase your branch on main and push it again? That should trigger it.

Done, everything seems to pass, thank you! 😃

@JPEWdev
Copy link
Collaborator

JPEWdev commented Jan 13, 2025

The signing tests where skipped for some reason, so I'm not sure the test run is valid. Can you please look into that?

@josch
Copy link
Author

josch commented Jan 13, 2025

Thank you, nice catch! I wonder though what is missing

I tried adding python3-gpg into .github/workflows/ci.yml apt-get installation line to no avail. The file pyproject.toml lists dependencies = [ "gpg >= 1.10.0" ] and the log says:

2025-01-13T18:59:32.8524042Z   Building wheel for gpg (pyproject.toml): started
2025-01-13T18:59:34.2209346Z   Building wheel for gpg (pyproject.toml): finished with status 'done'
2025-01-13T18:59:34.2213954Z   Created wheel for gpg: filename=gpg-1.10.0-cp312-cp312-linux_x86_64.whl size=40395 sha256=ed66083c743d0e3f2505316b7750464de7bfb979cef656163111a5e6503799fa
2025-01-13T18:59:34.2216864Z   Stored in directory: /home/runner/.cache/pip/wheels/7b/3e/c4/50fc15a2e3f7f5264dc8df81e6dec60449bf3f252808415713
2025-01-13T18:59:34.2236498Z Successfully built bmaptool gpg
2025-01-13T18:59:34.2461959Z Installing collected packages: gpg, six, platformdirs, pathspec, mypy-extensions, click, bmaptool, black
2025-01-13T18:59:34.5483116Z Successfully installed black-24.10.0 bmaptool-3.8.0 click-8.1.8 gpg-1.10.0 mypy-extensions-1.0.0

Still, in tests/test_CLI.py it fails to import the gpg module... I'll keep investigating...

@JPEWdev
Copy link
Collaborator

JPEWdev commented Jan 13, 2025

Maybe temporarily remove the try...except around the import and see if it gives a more useful error

@JPEWdev
Copy link
Collaborator

JPEWdev commented Jan 13, 2025

Ah right.... I attempt to fix this here: f61172a but it still doesn't work because it can't seem to import gpg after the change.

It's an unfortunate combination of the gpg and setuptools version.... and I'm not actually sure if there is a way to fix it cleanly since we are kinda stuck with version of gpg that comes from the distro.

@josch josch force-pushed the autogengpghome branch 3 times, most recently from 72e6599 to d489086 Compare January 14, 2025 06:24
@JPEWdev
Copy link
Collaborator

JPEWdev commented Jan 14, 2025

I was looking at your changes, and I finally figured out what the problem with the CI is. Basically, the pre-canned python versions provided by GitHub Actions can't possibly provide the gpg module correctly, because it must come from the host package. To facilitate this, I made a patch that extends the CI testing to test with "native" python, which can use the host python3-gpg, and thus run the tests. This should cherry-pick on your branch in place of the wip patch you have: JPEWdev@eb978ae and I'm pretty sure this will fix the CI

Thanks so much for looking into this, it was really helpful!

Fixes up the way that the GPG tests work by adding a new "native" python
test version. This is required because the python 'gpg' module *must*
come from the host package in order to patch libgpgme (e.g.
'python3-gpg'). It's not possible to get this module installed with the
pre-canned python versions provided by GitHub Actions, so the gpg tests
are skipped for this version, but using the host native python can.
@josch
Copy link
Author

josch commented Jan 16, 2025

Ah, sorry, I panic-removed my last comment. I must've clicked the wrong test because I still saw those skip messages. It looks fine for the native case now. Thanks! :)

@JPEWdev JPEWdev merged commit 15114f7 into yoctoproject:main Jan 16, 2025
7 checks passed
@JPEWdev
Copy link
Collaborator

JPEWdev commented Jan 16, 2025

Had to squash and merge because this repo only allows rebase or squash, and rebase wasn't an option because the branch was not up to date. Doesn't matter too much :)

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants