Proposal: more flexible argument extraction and signature testing

docs/Hook-Examples.md

@@ -25,7 +25,7 @@ although the examples on this page all use the JSON format.
 
 ## Incoming Github webhook
 
-This example works on 2.8+ versions of Webhook - if you are on a previous series, change `payload-hmac-sha1` to `payload-hash-sha1`.
+This example works on 2.9+ versions of Webhook - if you are on a previous series, change the `check-signature` block to an equivalent `match` block, see the [Hook-Rules](Hook-Rules.md#legacy-match-rules-for-signatures) page for full details.
 
 ```json
 [
@@ -53,11 +53,11 @@ This example works on 2.8+ versions of Webhook - if you are on a previous series
       "and":
       [
         {
-          "match":
+          "check-signature":
           {
-            "type": "payload-hmac-sha1",
+            "algorithm": "sha1",
             "secret": "mysecret",
-            "parameter":
+            "signature":
             {
               "source": "header",
               "name": "X-Hub-Signature"
@@ -185,11 +185,11 @@ Values in the request body can be accessed in the command or to the match rule b
       "and":
       [
         {
-          "match":
+          "check-signature":
           {
-            "type": "payload-hmac-sha256",
+            "algorithm": "sha256",
             "secret": "mysecret",
-            "parameter":
+            "signature":
             {
               "source": "header",
               "name": "X-Gogs-Signature"

docs/Hook-Rules.md

@@ -9,11 +9,12 @@
 * [Match](#match)
   * [Match value](#match-value)
   * [Match regex](#match-regex)
+  * [Match Whitelisted IP range](#match-whitelisted-ip-range)
+  * [Match scalr-signature](#match-scalr-signature)
+* [Check signature](#check-signature)
   * [Match payload-hmac-sha1](#match-payload-hmac-sha1)
   * [Match payload-hmac-sha256](#match-payload-hmac-sha256)
   * [Match payload-hmac-sha512](#match-payload-hmac-sha512)
-  * [Match Whitelisted IP range](#match-whitelisted-ip-range)
-  * [Match scalr-signature](#match-scalr-signature)
 
 ## And
 *And rule* will evaluate to _true_, if and only if all of the sub rules evaluate to _true_.
@@ -183,106 +184,120 @@ For the regex syntax, check out <http://golang.org/pkg/regexp/syntax/>
 }
 ```
 
-### Match payload-hmac-sha1
-Validate the HMAC of the payload using the SHA1 hash and the given *secret*.
+### Match Whitelisted IP range
+
+The IP can be IPv4- or IPv6-formatted, using [CIDR notation](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#CIDR_blocks).  To match a single IP address only, use `/32`.
+
 ```json
 {
   "match":
   {
-    "type": "payload-hmac-sha1",
-    "secret": "yoursecret",
-    "parameter":
-    {
-      "source": "header",
-      "name": "X-Hub-Signature"
-    }
+    "type": "ip-whitelist",
+    "ip-range": "192.168.0.1/24"
   }
 }
 ```
 
-Note that if multiple signatures were passed via a comma separated string, each
-will be tried unless a match is found. For example:
+Note this does not work if webhook is running behind a reverse proxy, as the "client IP" will either not be available at all (if webhook is using a Unix socket or named pipe) or it will be the address of the _proxy_, not of the real client.  You will probably need to enforce client IP restrictions in the reverse proxy itself, before forwarding the requests to webhook.
 
-```
-X-Hub-Signature: sha1=the-first-signature,sha1=the-second-signature
-```
+### Match scalr-signature
+
+The trigger rule checks the scalr signature and also checks that the request was signed less than 5 minutes before it was received. 
+A unique signing key is generated for each webhook endpoint URL you register in Scalr.
+Given the time check make sure that NTP is enabled on both your Scalr and webhook server to prevent any issues
 
-### Match payload-hmac-sha256
-Validate the HMAC of the payload using the SHA256 hash and the given *secret*.
 ```json
 {
   "match":
   {
-    "type": "payload-hmac-sha256",
-    "secret": "yoursecret",
-    "parameter":
-    {
-      "source": "header",
-      "name": "X-Signature"
-    }
+    "type": "scalr-signature",
+    "secret": "Scalr-provided signing key"
   }
 }
 ```
 
-Note that if multiple signatures were passed via a comma separated string, each
-will be tried unless a match is found. For example:
+## Check Signature
 
-```
-X-Hub-Signature: sha256=the-first-signature,sha256=the-second-signature
-```
+Many webhook protocols involve the hook sender computing an [HMAC](https://en.wikipedia.org/wiki/HMAC) _signature_ over the request content using a shared secret key, and sending the expected signature value as part of the webhook call.  The webhook recipient can then compute their own value for the signature using the same secret key and verify that value against the one supplied by the sender.  Since the sender and receiver are (or at least _should be_) the only parties that have knowledge of the secret, a matching signature guarantees that the payload is valid and was created by the legitimate sender.
+
+The `"check-signature"` rule type is used to validate these kinds of signatures.  In its simplest form you just specify the _algorithm_ (`sha1`, `sha256` or `sha512`), the _secret_, and where in the request to find the signature (typically a header or a query parameter).  Webhook will compute the HMAC over the whole of the request body using the supplied secret, and compare the result to the one taken from the request
 
-### Match payload-hmac-sha512
-Validate the HMAC of the payload using the SHA512 hash and the given *secret*.
 ```json
 {
-  "match":
+  "check-signature":
   {
-    "type": "payload-hmac-sha512",
+    "algorithm": "sha256",
     "secret": "yoursecret",
-    "parameter":
+    "signature":
     {
       "source": "header",
-      "name": "X-Signature"
+      "name": "X-Hub-Signature"
     }
   }
 }
 ```
 
 Note that if multiple signatures were passed via a comma separated string, each
-will be tried unless a match is found. For example:
+will be tried unless a match is found, and any `algorithm=` prefix is stripped off
+each signature value before comparison.  This allows for cases where the sender includes
+several signatures with different algorithms in the same header, e.g.:
 
 ```
-X-Hub-Signature: sha512=the-first-signature,sha512=the-second-signature
+X-Hub-Signature: sha1=the-sha1-signature,sha256=the-sha256-signature
 ```
 
-### Match Whitelisted IP range
+If the sender computes the signature over something other than just the request body then you can optionally provide a `"string-to-sign"` argument.  Usually this will be a template that assembles the string-to-sign from different parts of the request (one of which could be the body).  For example this would compute a signature over the values of the `X-Request-Id` header, `Date` header, and request body, separated by line breaks:
 
-The IP can be IPv4- or IPv6-formatted, using [CIDR notation](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#CIDR_blocks).  To match a single IP address only, use `/32`.
+```yaml
+check-signature:
+  algorithm: sha512
+  secret: 5uper5eecret
+  signature:
+    source: header
+    name: X-Hook-Signature
+  string-to-sign:
+    source: template
+    name: |
+      {{- printf "%s\r\n" (.GetHeader "x-request-id") -}}
+      {{- printf "%s\r\n" (.GetHeader "date") -}}
+      {{- .BodyText -}}
+```
+
+Note that signature algorithms can be very particular about whether "line breaks" are unix style LF or Windows-style CR+LF.  It is safest to be explicit, as in the above example, using `{{- -}}` blocks (that ignore the white space within the template itself either side of the block) and `printf` with `\n` or `\r\n`, to ensure the template generates the correct style of line endings whatever platform you created it on.
+
+### Legacy "match" rules for signatures
+In previous versions of webhook signature verification was handled by a set of specific "match" rule types named `payload-hmac-<algorithm>` - the legacy format is still understood but you may wish to update your existing configurations to the new format.
+
+The legacy configuration
 
 ```json
 {
   "match":
   {
-    "type": "ip-whitelist",
-    "ip-range": "192.168.0.1/24"
+    "type": "payload-hmac-<type>",
+    "secret": "secret",
+    "parameter":
+    {
+      "source": "header",
+      "name": "X-Signature"
+    }
   }
 }
 ```
 
-Note this does not work if webhook is running behind a reverse proxy, as the "client IP" will either not be available at all (if webhook is using a Unix socket or named pipe) or it will be the address of the _proxy_, not of the real client.  You will probably need to enforce client IP restrictions in the reverse proxy itself, before forwarding the requests to webhook.
-
-### Match scalr-signature
-
-The trigger rule checks the scalr signature and also checks that the request was signed less than 5 minutes before it was received. 
-A unique signing key is generated for each webhook endpoint URL you register in Scalr.
-Given the time check make sure that NTP is enabled on both your Scalr and webhook server to prevent any issues
+is equivalent to the new style
 
 ```json
 {
-  "match":
+  "check-signature":
   {
-    "type": "scalr-signature",
-    "secret": "Scalr-provided signing key"
+    "algorithm": "<type>",
+    "secret": "secret",
+    "signature":
+    {
+      "source": "header",
+      "name": "X-Signature"
+    }
   }
 }
 ```

docs/Referencing-Request-Values.md

@@ -132,3 +132,79 @@ and for query variables you can use
   "source": "entire-query"
 }
 ```
+
+# Using a template
+If the above source types do not provide sufficient flexibility for your needs, it is possible to provide a [Go template][tt] to compute the value.  The template _context_ provides access to the headers, query parameters, parsed payload, and the complete request body content.  For clarity, the following examples show the YAML form of the definition rather than JSON, since template strings will often contain double quotes, line breaks, etc. that need to be specially encoded in JSON.
+
+## Examples
+
+Extract a value from the payload, if it is present, otherwise from the query string (this allows for a hook that may be called with either a POST request with the form data in the payload, or a GET request with the same data in the URL):
+
+```yaml
+- source: template
+  name: |-
+    {{- with .Payload.requestId -}}
+      {{- . -}}
+    {{- else -}}
+      {{- .Query.requestId -}}
+    {{- end -}}
+```
+
+Given the following JSON payload describing multiple commits:
+
+```json
+{
+  "commits": [
+    {
+      "commit": {
+        "commit-id": 1
+      }
+    }, {
+      "commit": {
+        "commit-id": 2
+      }
+    }
+  ]
+}
+```
+
+this template would generate a semicolon-separated list of all the commit IDs:
+
+```yaml
+- source: template
+  name: |-
+    {{- range $i, $c := .Payload.commits -}}
+      {{- if gt $i 0 -}};{{- end -}}
+      {{- index $c.commit "commit-id" -}}
+    {{- end -}}
+```
+
+Here `.Payload.commits` is the array of objects, each of these has a field `commit`, which in turn has a field `commit-id`.  The `range` operator iterates over the commits array, setting `$i` to the (zero-based) index and `$c` to the object.  The template then prints a semicolon if this is not the first iteration, then we extract the `commit` field from that object, then in turn the `commit-id`.  Note how the first level can be extracted with just `$c.commit` because the field name is a valid identifier, but for the second level we must use the `index` function.
+
+To access request _header_ values, use the `.GetHeader` function:
+
+```yaml
+- source: template
+  name: |-
+    {{- .GetHeader "x-request-id" }}:{{ index .Query "app-id" -}}
+```
+
+## Template context
+
+The following items are available to templates, in addition to the [standard functions](https://pkg.go.dev/text/template#hdr-Functions) provided by Go:
+
+- `.Payload` - the parsed request payload, which may be JSON, XML or form data.
+- `.Query` - the query string parameters from the hook URL.
+- `.GetHeader "header-name"` - function that returns the value of the given request header, case-insensitive
+- `.ContentType` - the request content type
+- `.ID` - request ID assigned by webhook itself
+- `.Method` - the HTTP request method (`GET`, `POST`, etc.)
+- `.RemoteAddr` - IP address of the client (though this may not be accurate if webhook is behind a [reverse proxy](Hook-Rules.md#match-whitelisted-ip-range))
+- `.BodyText` - the complete raw content of the request body, as a string
+
+The following are also available but less frequently needed:
+
+- `.Body` - complete body content, but as a slice of bytes rather than as a string
+- `.Headers` - the map of HTTP headers.  Useful if you need to `range` over the headers, but to look up keys directly in this map you must use the canonical form - the `.GetHeader` function performs a case-insensitive lookup.
+
+[tt]: https://golang.org/pkg/text/template/

docs/Templates.md

@@ -73,5 +73,33 @@ Additionally, the result is piped through the built-in Go template function `js`
 
 ```
 
+## Changing the template delimiters
+
+If your hook configuration includes lookup arguments of type `{"source": "template"}`, and you also need to parse the hooks file _as_ a template, you can use the `-template-delims` parameter to change the template delimiter used when processing the hook file so it does not clash with the standard `{{ ... }}` delimiters used for template lookups.  The parameter is a comma-separated pair of the left and right delimiter strings, e.g. `-template-delims='[[,]]'` would use square brackets.  For a configuration like this:
+
+```json
+[
+  {
+    "id": "example",
+    "trigger-rule": {
+      "check-signature": {
+        "algorithm": "sha256",
+        "secret": "[[ getenv `XXXTEST_SECRET` | js ]]",
+        "signature": {
+          "source": "header",
+          "name": "X-Signature"
+        },
+        "string-to-sign": {
+          "source": "template",
+          "name": "{{ .BodyText }}{{ .GetHeader `date` }}"
+        }
+      }
+    }
+  }
+]
+```
+
+the `-template-delims='[[,]]'` setting would cause the `getenv` part to be interpreted when parsing the hook file, whereas the string-to-sign template would be executed when evaluating the trigger rule against each request.
+
 [w]: https://github.com/adnanh/webhook
 [tt]: https://golang.org/pkg/text/template/