Webhook Signatures and Validation

[EvanMerlock]

What problem would you like to solve? Please describe:
It should be possible to validate that a given webhook payload came from a specific GoAlert instance and that it wasn't tampered with over the wire.

Describe the solution you'd like:
By signing a webhook payload and sending that signature as an HTTP header, we can validate that (so long as the transport protocol is secure, e.g. HTTPS) a webhook payload came from a specific GoAlert instance. This would also ensure tampering doesn't happen.

Describe alternatives you've considered:
Including a bearer token, basic auth, or other authN/Z protocol as part of the webhook delivery mechanism would give stronger guarantees than "this payload came from this GoAlert instance" (e.g. "this payload came from this GoAlert delivery method") but would involve potentially storing credentials for each webhook delivery method.

Similarly, having signing keys for each webhook delivery method (per user, etc) would cause the same "credential bloat" but would avoid the risk of end-user credential rotation (like basic AuthN/Z or another form of end-user provided credential solution). I think it is a safe assumption to make that an end-user should "trust" all webhooks originating from a given GoAlert instance. Whether or not the application receiving the webhook should act upon a received webhook once it has been validated is likely not a concern of GoAlert which sends the webhook.

Additional context:
Sketching out a solution, it is likely that the simplest approach is to use a single Keyring per instance that can be manually rotated via the API and sign the webhook as it gets delivered. The public key could be accessed via the UI or API (proof that it "came from this specific GoAlert instance") and used by the receiving party to validate. I'm going to try writing this up.

[mastercactapus]

This makes sense. The primary problem is "proving" that a request came from a trusted application. Webhook URLs can currently contain credentials, which can address some of the concerns, including per-config auth.

What's left is knowing where it came from, and a public/private key approach would work well with that, as any receiving application can then verify.

As for implementation, maybe break this into three parts:

1. Add a new non-auto-rotating keyring, set a header, add a smoketest to validate
2. Expose public key and how-to-validate in the UI
3. For rotation, since the secret is never shared -- and rotating would break everything; is it necessary? Instead, it may make sense to have a top-level GoAlert option that is "reset ALL keyrings" in case of database + encryption key compromise but where you want to keep your config. For webhook signatures alone, since no secret is ever shared, or unencrypted at rest, I'm not sure there's a scenario where a rotation would be necessary that wouldn't also include all session tokens, API keys, etc...

[EvanMerlock]

For rotation, since the secret is never shared -- and rotating would break everything; is it necessary?

I tend to agree it is unnecessary to be able to individually rotate the single key-ring...

Instead, it may make sense to have a top-level GoAlert option that is "reset ALL keyrings" in case of database + encryption key compromise but where you want to keep your config.

Agreed. Not sure it's a prerequisite of this functionality, it'd be important either way

[mastercactapus]

I did some digging around existing examples, this is the most active/comprehensive RFC I could find for existing work solving this problem in the wild:
https://datatracker.ietf.org/doc/rfc9421/

Looks like there's a tool out there to experiment with a list of libraries:
https://httpsig.org/

TL;DR; uses two headers:

• Signature-Input which specifies things like which headers to use in the calculation (incl. special ones for @method and @target-uri)
• Signature with the actual signature. The body payload is included by including a Content-Digest header with the request and then including that with the signature.

It's also possible to specify multiple signatures, which would allow for key rotation or even config changes/deprecations as necessary in the future.

An example of what a request with two signatures might look like

POST /path?param=value HTTP/1.1
Host: www.example.com
Date: Tue, 20 Apr 2021 02:07:55 GMT
Content-Type: application/json
Content-Digest: sha-512=:WZDPaVn/7XgHaAy8pmojAkGWoRx2UFChF41A2svX+TaPm+AbwAgBWnrIiYllu7BNNyealdVLvRwEmTHWXvJwew==:
Content-Length: 18
Signature-Input: sig1=("@method" "@target-uri" "content-digest");created=1618884473;keyid="test-key-old";alg="ecdsa-p256-sha256"
Signature: sig1=:dMT/A/76ehrdBTD/2Xx8QuKV6FoyzEP/I9hdzKN8LQJLNgzU4W767HK05rx1i8meNQQgQPgQp8wq2ive3tV5Ag==:
Signature-Input: sig2=("@method" "@target-uri" "content-digest");created=1618884475;keyid="test-key-new";alg="ecdsa-p256-sha256"
Signature: sig2=:C73J41GVKc+TYXbSobvZf0CmNcptRiWN+NY1Or0A36ISg6ymdRN6ZgR2QfrtopFNzqAyv+CeWrMsNbcV2Ojsgg==:

{"hello": "world"}

In this example:

• There are two sets of Signature-Input and Signature fields, one for "sig1" and one for "sig2."
• The keyid in "sig1" refers to the "test-key-old" key, and "sig2" refers to the "test-key-new" key.
• Both signatures cover the same components (@method, @target-uri, content-digest), but you can choose different sets of covered components for each signature if necessary.
• The created parameter shows different timestamps, but the timestamps could be the same if you have a use case for that.

---

As for what we'd use, ECDSA with SHA-256 and P-256 is becoming widely accepted, so I think that would be a good balance. Also, if GoAlert calls into any environments/applications that need to be FIPS compliant, there are options for that combo.

Lastly, for how/where to put this, we could extend the existing keyring to support overriding its curve and adding a new method if necessary to sign; otherwise we could consider making a new one that supports more flexibility around keys.

The existing code is a bit dated, so it may be good to update it one way or another (in-place or replace).

[EvanMerlock]

If I understand that RFC correctly, the actual signature would not be generated strictly from the message body itself but from a "signature base" derived from the contents of the message.

The yaronf/httpsign package includes basic middleware + support for all signers, etc, which seems promising (depending on our dependency posture).

I am not entirely comfortable with ECDSA/SHA256/P256, but I think the majority of possible attacks are timing related. I am outside of my depth here though... cryptography is not my strong suit. It is a supported algorithm in RFC9421. If we used yarnof/httpsign, I think we'd move away from Keyring as signer and instead keep it only as Keyring as storage, meaning we'd only need to implement overriding the curve

If we are to use RFC9421, which seems wise, we could very easily sign every outgoing GoAlert request with middleware if desired (and it likely is).

My biggest concerned with RFC9421 is how many implementation libraries exist for each language, but rolling our own scheme seems more fraught with error, even if it is simpler than the RFC.

[EvanMerlock]

Speaking on potential JWK endpoints, we could use /.well-known/jwks.json as a list of all known public keys (including revocation / rotation status) and use a library to encode ecdsa.PublicKey instances to JWKs for output, noting that we'd need to use ECDSA-P256 or Ed25519.

I think we'd want to provide the current public key in the UI as a PEM-encoded blob, which is likely the most compatible format for future loading (or at least is the easiest to translate to different formats). This could be accomplished via the GQL API, presenting the current public key as a portion of the schema.