Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LibCrypto: RSA encryption/decryption and modes with OpenSSL #3234

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

devgianlu
Copy link
Contributor

This PR is where OpenSSL comes to the rescue by letting us replace our own implementation of RSA-PCKS1 and RSA-OAEP with a few lines of code (mind all the framework around it 😄).

The commit separation isn't the best because the new RSA::verify and RSA::sign methods are not compatible with the old padding scheme classes so everything had to be replaced at once.

It used to be that the caller would supply a buffer to write the output
to. This created an anti-pattern in multiple places where the caller
would allocate a `ByteBuffer` and then use `.bytes()` to provide it to
the `PKSystem` method. Then the callee would resize the output buffer
and reassign it, but because the resize was on `Bytes` and not on
`ByteBuffer`, the caller using the latter would cause a bug.

Additionally, in pretty much all cases the buffer was pre-allocated
shortly before.
@devgianlu devgianlu requested a review from alimpfard as a code owner January 12, 2025 11:17
@alimpfard
Copy link
Contributor

I took a very brief look over the changes and they seem reasonable (actual review pending though), openssl sure hasn't gotten any nicer to look at.

@devgianlu devgianlu force-pushed the rework-rsa branch 2 times, most recently from 99278ee to 7c4854e Compare January 12, 2025 16:57
Add a forwarding header for OpenSSL types so that we can build without
propagating the OpenSSL dependency.
This is a small change to allow subclasses of `RSA` to configure the
`EVP_PKEY_CTX` without rewriting everything.
This serve as a regression check for the next commits
This commit replaces the old implementation of `EMSA_PKCS1_V1_5` with
one backed by OpenSSL. In doing so, the `sign` and `verify` methods of
RSA have been modified to behave like expected and not just be
encryption and decryption.

I was not able to split this commit because the changes to `verify` and
`sign` break pretty much everything.
This replaces the old `OAEP` implementation with one backed by OpenSSL.
The changes also include some added modularity to the RSA class by
making the `RSA_EME` and `RSA_EMSE` for encryption/decryption and
signing/verifying respectively.
@devgianlu
Copy link
Contributor Author

Fixed the macOS build issue. If this got too big to review let me know and I'll make a separate one for the build things.

The CI failure is for a flaky test. @alimpfard Can you restart that single job?

openssl sure hasn't gotten any nicer to look at

I've started to get a feeling for it, the documentation of the single methods is usually pretty decent, but it lacks in explaining how to use the things together. I've been looking at the openssl CLI code and GitHub search a lot.

Once most of the stuff is in place we can work on abstracting the most common patterns into the wrappers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants