Providers

The Provider API defines how events are encoded and where to send them.

Example

The following is an example of how to send alerts to Slack when Flux fails to install or upgrade Flagger.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack-bot
  namespace: flagger-system
spec:
  type: slack
  channel: general
  address: https://slack.com/api/chat.postMessage
  secretRef:
    name: slack-bot-token
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: slack
  namespace: flagger-system
spec:
  summary: "Flagger impacted in us-east-2"
  providerRef:
    name: slack-bot
  eventSeverity: error
  eventSources:
    - kind: HelmRepository
      name: '*'
    - kind: HelmRelease
      name: '*'

In the above example:

A Provider named slack-bot is created, indicated by the Provider.metadata.name field.
An Alert named slack is created, indicated by the Alert.metadata.name field.
The Alert references the slack-bot provider, indicated by the Alert.spec.providerRef field.
The notification-controller starts listening for events sent for all HelmRepositories and HelmReleases in the flagger-system namespace.
When an event with severity error is received, the controller posts a message on Slack containing the summary text and the Helm install or upgrade error.
The controller uses the Slack Bot token from the secret indicated by the Provider.spec.secretRef.name to authenticate with the Slack API.

You can run this example by saving the manifests into slack-alerts.yaml.

First create a secret with the Slack bot token:

kubectl -n flagger-system create secret generic slack-bot-token --from-literal=token=xoxb-YOUR-TOKEN

Apply the resources on the cluster:

kubectl -n flagger-system apply --server-side -f slack-alerts.yaml

Writing a provider spec

As with all other Kubernetes config, a Provider needs apiVersion, kind, and metadata fields. The name of an Alert object must be a valid DNS subdomain name.

A Provider also needs a .spec section.

Type

.spec.type is a required field that specifies which SaaS API to use.

The supported alerting providers are:

Provider	Type
Generic webhook	`generic`
Generic webhook with HMAC	`generic-hmac`
Azure Event Hub	`azureeventhub`
DataDog	`datadog`
Discord	`discord`
GitHub dispatch	`githubdispatch`
Google Chat	`googlechat`
Google Pub/Sub	`googlepubsub`
Grafana	`grafana`
Lark	`lark`
Matrix	`matrix`
Microsoft Teams	`msteams`
Opsgenie	`opsgenie`
PagerDuty	`pagerduty`
Prometheus Alertmanager	`alertmanager`
Rocket	`rocket`
Sentry	`sentry`
Slack	`slack`
Telegram	`telegram`
WebEx	`webex`
NATS	`nats`

The supported providers for Git commit status updates are:

Provider	Type
Azure DevOps	`azuredevops`
Bitbucket	`bitbucket`
Bitbucket Server/Data Center	`bitbucketserver`
GitHub	`github`
GitLab	`gitlab`
Gitea	`gitea`

Alerting

Generic webhook

When .spec.type is set to generic, the controller will send an HTTP POST request to the provided Address.

The body of the request is a JSON Event object, for example:

{
  "involvedObject": {
    "apiVersion": "kustomize.toolkit.fluxcd.io/v1",
    "kind": "Kustomization",
    "name": "webapp",
    "namespace": "apps",
    "uid": "7d0cdc51-ddcf-4743-b223-83ca5c699632"
  },
  "metadata": {
    "kustomize.toolkit.fluxcd.io/revision": "main/731f7eaddfb6af01cb2173e18f0f75b0ba780ef1"
  },
  "severity":"error",
  "reason": "ValidationFailed",
  "message":"service/apps/webapp validation error: spec.type: Unsupported value: Ingress",
  "reportingController":"kustomize-controller",
  "reportingInstance":"kustomize-controller-7c7b47f5f-8bhrp",
  "timestamp":"2022-10-28T07:26:19Z"
}

Where the involvedObject key contains the metadata from the object triggering the event.

The controller includes a Gotk-Component header in the request, which can be used to identify the component which sent the event, e.g. source-controller or notification-controller.

POST / HTTP/1.1
Host: example.com
Accept-Encoding: gzip
Content-Length: 452
Content-Type: application/json
Gotk-Component: kustomize-controller
User-Agent: Go-http-client/1.1

You can add additional headers to the POST request using a headers key in the referenced Secret.

Generic webhook with HMAC

When .spec.type is set to generic-hmac, the controller will send an HTTP POST request to the provided Address for an Event, while including an X-Signature HTTP header carrying the HMAC of the request body. The inclusion of the header allows the receiver to verify the authenticity and integrity of the request.

The X-Signature header is calculated by generating an HMAC of the request body using the token key from the referenced Secret. The HTTP header value has the following format:

X-Signature: <hash-function>=<hash>

<hash-function> denotes the hash function used to generate the HMAC and currently defaults to sha256, which may change in the future. <hash> is the HMAC of the request body, encoded as a hexadecimal string.

while <hash> is the hex-encoded HMAC value.

The body of the request is a JSON Event object, as described in the Generic webhook section.

HMAC verification example

The following example in Go shows how to verify the authenticity and integrity of a request by using the X-Signature header.

func verifySignature(signature string, payload, key []byte) error {
	sig := strings.Split(signature, "=")

	if len(sig) != 2 {
		return fmt.Errorf("invalid signature value")
	}

	var newF func() hash.Hash
	switch sig[0] {
	case "sha224":
		newF = sha256.New224
	case "sha256":
		newF = sha256.New
	case "sha384":
		newF = sha512.New384
	case "sha512":
		newF = sha512.New
	default:
		return fmt.Errorf("unsupported signature algorithm %q", sig[0])
	}

	mac := hmac.New(newF, key)
	if _, err := mac.Write(payload); err != nil {
		return fmt.Errorf("failed to write payload to HMAC encoder: %w", err)
	}

	sum := fmt.Sprintf("%x", mac.Sum(nil))
	if sum != sig[1] {
		return fmt.Errorf("HMACs do not match: %#v != %#v", sum, sig[1])
	}
	return nil
}

func handleRequest(w http.ResponseWriter, r *http.Request) {
	// Require a X-Signature header
	if len(r.Header["X-Signature"]) == 0 {
		http.Error(w, "missing X-Signature header", http.StatusBadRequest)
		return
	}

	// Read the request body with a limit of 1MB
	lr := io.LimitReader(r.Body, 1<<20)
	body, err := io.ReadAll(lr)
	if err != nil {
		http.Error(w, "failed to read request body", http.StatusBadRequest)
		return
	}

	// Verify signature using the same token as the Secret referenced in
	// Provider
	key := []byte("<token>")
	if err := verifySignature(r.Header.Get("X-Signature"), body, key); err != nil {
		http.Error(w, fmt.Sprintf("failed to verify HMAC signature: %s", err.Error()), http.StatusBadRequest)
		return
	}

	// Do something with the verified request body
	// ...
}

Slack

When .spec.type is set to slack, the controller will send a message for an Event to the provided Slack API Address.

The Event will be formatted into a Slack message using an Attachment, with the metadata attached as fields, and the involved object as author. The severity of the Event is used to set the color of the attachment.

When a Channel is provided, it will be added as a channel field to the API payload. Otherwise, the further configuration of the Address will determine the channel.

When Username is set, this will be added as a username field to the payload, defaulting to the name of the reporting controller.

This Provider type supports the configuration of a proxy URL and/or TLS certificates.

Slack example

To configure a Provider for Slack, we recommend using a Slack Bot App token which is not attached to a specific Slack account. To obtain a token, please follow Slack's guide on creating an app.

Once you have obtained a token, create a Secret containing the token key and a slack Provider with the address set to https://slack.com/api/chat.postMessage.

Using this API endpoint, it is possible to send messages to multiple channels by adding the integration to each channel.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: default
spec:
  type: slack
  channel: general
  address: https://slack.com/api/chat.postMessage
  secretRef:
    name: slack-token
---
apiVersion: v1
kind: Secret
metadata:
  name: slack-token
  namespace: default
stringData:
    token: xoxb-1234567890-1234567890-1234567890-1234567890

Slack (legacy) example

To configure a Provider for Slack using the legacy incoming webhook API, create a Secret with the address set to https://hooks.slack.com/services/..., and a slack Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: default
spec:
  type: slack
  secretRef:
    name: slack-webhook
---
apiVersion: v1
kind: Secret
metadata:
  name: slack-webhook
  namespace: default
stringData:
  address: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"

Microsoft Teams

When .spec.type is set to msteams, the controller will send a payload for an Event to the provided Address. The address may be a Microsoft Teams Incoming Webhook Workflow, or the deprecated Office 365 Connector.

Note: If the Address host contains the suffix .webhook.office.com, the controller will imply that the backend is the deprecated Office 365 Connector and is expecting the Event in the connector message format. Otherwise, the controller will format the Event as a Microsoft Adaptive Card message.

In both cases the Event metadata is attached as facts, and the involved object as a summary/title. The severity of the Event is used to set the color of the message.

This Provider type supports the configuration of a proxy URL and/or TLS certificates, but lacks support for configuring a Channel. This can be configured during the creation of the Incoming Webhook Workflow in Microsoft Teams.

Microsoft Teams example

To configure a Provider for Microsoft Teams, create a Secret with the address set to the webhook URL, and an msteams Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: msteams
  namespace: default
spec:
  type: msteams
  secretRef:
    name: msteams-webhook
---
apiVersion: v1
kind: Secret
metadata:
  name: msteams-webhook
  namespace: default
stringData:
  address: https://prod-xxx.yyy.logic.azure.com:443/workflows/zzz/triggers/manual/paths/invoke?...

DataDog

When .spec.type is set to datadog, the controller will send a payload for an Event to the provided DataDog API Address.

The Event will be formatted into a DataDog Event and sent to the API endpoint of the provided DataDog Address.

This Provider type supports the configuration of a proxy URL and/or TLS certificates.

The metadata of the Event is included in the DataDog event as extra tags.

DataDog example

To configure a Provider for DataDog, create a Secret with the token set to a DataDog API key (not an application key!) and a datadog Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: datadog
  namespace: default
spec:
  type: datadog
  address: https://api.datadoghq.com # DataDog Site US1
  secretRef:
    name: datadog-secret
---
apiVersion: v1
kind: Secret
metadata:
  name: datadog-secret
  namespace: default
stringData:
  token: <DataDog API Key>
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: datadog-info
  namespace: default
spec:
  eventSeverity: info
  eventSources:
    - kind: HelmRelease
      name: "*"
  providerRef:
    name: datadog
  eventMetadata:
    env: my-k8s-cluster # example of adding a custom `env` tag to the event

Discord

When .spec.type is set to discord, the controller will send a payload for an Event to the provided Discord Address.

The Event will be formatted into a Slack message and send to the /slack endpoint of the provided Discord Address.

This Provider type supports the configuration of a proxy URL and/or TLS certificates, but lacks support for configuring a Channel. This can be configured during the creation of the address

Discord example

To configure a Provider for Discord, create a Secret with the address set to the webhook URL, and a discord Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: discord
  namespace: default
spec:
  type: discord
  secretRef:
    name: discord-webhook
---
apiVersion: v1
kind: Secret
metadata:
  name: discord-webhook
  namespace: default
stringData:
    address: "https://discord.com/api/webhooks/..."

Sentry

When .spec.type is set to sentry, the controller will send a payload for an Event to the provided Sentry Address.

Depending on the severity of the Event, the controller will capture a Sentry Eventfor error, or Sentry Transaction Event with a Span for info. The metadata of the Event is included as extra data in the Sentry Event, or as Span tags.

The Provider's Channel is used to set the environment on the Sentry client.

This Provider type supports the configuration of TLS certificates.

Sentry example

To configure a Provider for Sentry, create a Secret with the address set to a Sentry DSN, and a sentry Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: sentry
  namespace: default
spec:
  type: sentry
  channel: staging-env
  secretRef:
    name: sentry-webhook
---
apiVersion: v1
kind: Secret
metadata:
  name: sentry-webhook
  namespace: default
stringData:
  address: "https://....@sentry.io/12341234"

Note: The sentry Provider also sends traces for events with the severity info. This can be disabled by setting the spec.eventSeverity field to error on an Alert.

Telegram

When .spec.type is set to telegram, the controller will send a payload for an Event to the provided Telegram Address.

The Event will be formatted into a message string, with the metadata attached as a list of key-value pairs.

The Provider's Channel is used to set the receiver of the message. This can be a unique identifier (-1234567890) for the target chat, or the username (@username) of the target channel.

This Provider type does not support the configuration of a proxy URL or TLS certificates.

Telegram example

To configure a Provider for Telegram, create a Secret with the token obtained from the BotFather, and a telegram Provider with a Secret reference, and the address set to https://api.telegram.org.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: telegram
  namespace: default
spec:
  type: telegram
  address: https://api.telegram.org
  channel: "@fluxcd" # or "-1557265138" (channel id)
  secretRef:
    name: telegram-token

Matrix

When .spec.type is set to matrix, the controller will send a payload for an Event to the provided Matrix Address.

The Event will be formatted into a message string, with the metadata attached as a list of key-value pairs, and send as a m.room.message text event to the provided Matrix Address.

The Provider's Channel is used to set the receiver of the message using a room identifier (!1234567890:example.org).

This provider type does support the configuration of TLS certificates.

Matrix example

To configure a Provider for Matrix, create a Secret with the token obtained from the Matrix endpoint, and a matrix Provider with a Secret reference.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: matrix
  namespace: default
spec:
  type: matrix
  address: https://matrix.org
  channel: "!jezptmDwEeLapMLjOc:matrix.org"
  secretRef:
    name: matrix-token

Lark

When .spec.type is set to lark, the controller will send a payload for an Event to the provided Lark Address.

The Event will be formatted into a Lark Message card, with the metadata written to the message string.

This Provider type does not support the configuration of a proxy URL or TLS certificates.

Lark example

To configure a Provider for Lark, create a Secret with the address obtained from adding a bot to a group, and a lark Provider with a Secret reference.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: lark
  namespace: default
spec:
  type: lark
  secretRef:
    name: lark-webhook
---
apiVersion: v1
kind: Secret
metadata:
  name: lark-webhook
  namespace: default
stringData:
    address: "https://open.larksuite.com/open-apis/bot/v2/hook/xxxxxxxxxxxxxxxxx"

Rocket

When .spec.type is set to rocket, the controller will send a payload for an Event to the provided Rocket Address.

The Event will be formatted into a Slack message and send as a payload the provided Rocket Address.

This Provider type does support the configuration of a proxy URL and TLS certificates.

Rocket example

To configure a Provider for Rocket, create a Secret with the address set to the Rocket webhook URL, and a rocket Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: rocket
  namespace: default
spec:
  type: rocket
  secretRef:
    name: rocket-webhook

Google Chat

When .spec.type is set to googlechat, the controller will send a payload for an Event to the provided Google Chat Address.

The Event will be formatted into a Google Chat card message, with the metadata added as a list of key-value pairs in a widget.

This Provider type does support the configuration of a proxy URL.

Google Chat example

To configure a Provider for Google Chat, create a Secret with the address set to the Google Chat webhook URL, and a googlechat Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: google
  namespace: default
spec:
  type: googlechat
  secretRef:
    name: google-webhook
---
apiVersion: v1
kind: Secret
metadata:
  name: google-webhook
  namespace: default
stringData:
  address: https://chat.googleapis.com/v1/spaces/...

Google Pub/Sub

When .spec.type is set to googlepubsub, the controller will publish the payload of an Event on the Google Pub/Sub Topic ID provided in the Channel field, which must exist in the GCP Project ID provided in the Address field.

This Provider type can optionally use the Secret reference to authenticate on the Google Pub/Sub API by using JSON credentials. The credentials must be specified on the token field of the Secret.

If no JSON credentials are specified, then the automatic authentication methods of the Google libraries will take place, and therefore methods like Workload Identity will be automatically attempted.

The Google identity effectively used for publishing messages must have the required permissions on the Pub/Sub Topic.

You can optionally add attributes to the Pub/Sub message using a headers key in the referenced Secret.

This Provider type does not support the configuration of a proxy URL or TLS certificates.

Google Pub/Sub with JSON Credentials and Custom Headers Example

To configure a Provider for Google Pub/Sub authenticating with JSON credentials and custom headers, create a Secret with the token set to the necessary JSON credentials, the headers field set to a YAML string-to-string dictionary, and a googlepubsub Provider with the associated Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: googlepubsub-provider
  namespace: desired-namespace
spec:
  type: googlepubsub
  address: <GCP Project ID>
  channel: <Pub/Sub Topic ID>
  secretRef:
    name: googlepubsub-provider-creds
---
apiVersion: v1
kind: Secret
metadata:
  name: googlepubsub-provider-creds
  namespace: desired-namespace
stringData:
  token: <GCP JSON credentials>
  headers: |
    attr1-name: attr1-value
    attr2-name: attr2-value

Opsgenie

When .spec.type is set to opsgenie, the controller will send a payload for an Event to the provided Opsgenie Address.

The Event will be formatted into a Opsgenie alert, with the metadata added to the details field as a list of key-value pairs.

This Provider type does support the configuration of a proxy URL and TLS certificates.

Opsgenie example

To configure a Provider for Opsgenie, create a Secret with the token set to the Opsgenie API token, and a opsgenie Provider with a Secret reference and the address set to https://api.opsgenie.com/v2/alerts.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: opsgenie
  namespace: default
spec:
  type: opsgenie
  address: https://api.opsgenie.com/v2/alerts
  secretRef:
    name: opsgenie-token
---
apiVersion: v1
kind: Secret
metadata:
  name: opsgenie-token
  namespace: default
stringData:
    token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

PagerDuty

When .spec.type is set to pagerduty, the controller will send a payload for an Event to the provided PagerDuty Address.

The Event will be formatted into an Event API v2 payload, triggering or resolving an incident depending on the event's Severity.

The provider will also send Change Events for info level Severity, which will be displayed in the PagerDuty service's timeline to track changes.

This Provider type supports the configuration of a proxy URL and TLS certificates.

The Channel is used to set the routing key to send the event to the appropriate integration.

PagerDuty example

To configure a Provider for Pagerduty, create a pagerduty Provider, set address to the integration URL and channel set to the integration key (also known as a routing key) for your service or event orchestration.

When adding an integration for a service on PagerDuty, it is recommended to use Events API v2 integration.

Note: PagerDuty does not support Change Events when sent to global integrations, such as event orchestration.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: pagerduty
  namespace: default
spec:
  type: pagerduty
  address: https://events.pagerduty.com
  channel: <integrationKey>

If you are sending to a service integration, it is recommended to set your Alert to filter to only those sources you want to trigger an incident for that service. For example:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: my-service-pagerduty
  namespace: default
spec:
  providerRef:
    name: pagerduty
  eventSources:
    - kind: HelmRelease
      name: my-service
      namespace: default

Prometheus Alertmanager

When .spec.type is set to alertmanager, the controller will send a payload for an Event to the provided Prometheus Alertmanager Address.

The Event will be formatted into a firing Prometheus Alertmanager alert, with the metadata added to the labels fields, and the message (and optional .metadata.summary) added as annotations. Event timestamp will be used to set alert start time (.StartsAt).

In addition to the metadata from the Event, the following labels will be added:

Label	Description
alertname	The string Flux followed by the Kind and the reason for the event e.g `FluxKustomizationProgressing`
severity	The severity of the event (`error` or `info`)
reason	The machine readable reason for the objects transition into the current status
kind	The kind of the involved object associated with the event
name	The name of the involved object associated with the event
namespace	The namespace of the involved object associated with the event

Note that due to the way other Flux controllers currently emit events, there's no way for notification-controller to figure out the time the event ends to set .EndsAt (a reasonable estimate being double the reconciliation interval of the resource involved) that doesn't involve a Kubernetes API roundtrip. A possible workaround could be setting global.resolve_timeout to an interval large enough for events to reoccur:

global:
  resolve_timeout: 1h

This Provider type does support the configuration of a proxy URL and TLS certificates.

Prometheus Alertmanager example

To configure a Provider for Prometheus Alertmanager, create a Secret with the address set to the Prometheus Alertmanager HTTP API URL including Basic Auth credentials, and a alertmanager Provider with a Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: alertmanager
  namespace: default
spec:
  type: alertmanager
  secretRef:
    name: alertmanager-address
---
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-address
  namespace: default
stringData:
    address: https://username:password@<alertmanager-url>/api/v2/alerts/"

Webex

When .spec.type is set to webex, the controller will send a payload for an Event to the provided Webex Address.

The Event will be formatted into a message string, with the metadata attached as a list of key-value pairs, and send as a Webex message.

The Channel is used to set the ID of the room to send the message to.

This Provider type does support the configuration of a proxy URL and TLS certificates.

Webex example

To configure a Provider for Webex, create a Secret with the token set to the Webex access token, and a webex Provider with a Secret reference and the address set to https://webexapis.com/v1/messages.

Note: To be able to send messages to a Webex room, the bot needs to be added to the room. Failing to do so will result in 404 errors, logged by the controller.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: webex
  namespace: default
spec:
  type: webex
  address: https://webexapis.com/v1/messages
  channel: <webexSpaceRoomID>
  secretRef:
    name: webex-token
---
apiVersion: v1
kind: Secret
metadata:
  name: webex-token
  namespace: default
stringData:
  token: <bot-token>

NATS

When .spec.type is set to nats, the controller will publish the payload of an Event on the NATS Subject provided in the Channel field, using the server specified in the Address field.

This Provider type can optionally use the Secret reference to authenticate to the NATS server using Username/Password. The credentials must be specified in the username and password fields of the Secret. Alternatively, NATS also supports passing the credentials with the server URL. In this case the address should be provided through a Secret reference.

Additionally if using credentials, the User must have authorization to publish on the Subject provided.

NATS with Username/Password Credentials Example

To configure a Provider for NATS authenticating with Username/Password, create a Secret with the username and password fields set, and add a nats Provider with the associated Secret reference.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: nats-provider
  namespace: desired-namespace
spec:
  type: nats
  address: <NATS Server URL>
  channel: <Subject>
  secretRef:
    name: nats-provider-creds
---
apiVersion: v1
kind: Secret
metadata:
  name: nats-provider-creds
  namespace: desired-namespace
stringData:
  username: <NATS Username>
  password: <NATS Password>

Address

.spec.address is an optional field that specifies the endpoint where the events are posted. The meaning of endpoint here depends on the specific Provider type being used. For the generic Provider for example this is an HTTP/S address. For other Provider types this could be a project ID or a namespace.

If the address contains sensitive information such as tokens or passwords, it is recommended to store the address in the Kubernetes secret referenced by .spec.secretRef.name. When the referenced Secret contains an address key, the .spec.address value is ignored.

Channel

.spec.channel is an optional field that specifies the channel where the events are posted.

Username

.spec.username is an optional field that specifies the username used to post the events. Can be overwritten with a Secret reference.

Secret reference

.spec.secretRef.name is an optional field to specify a name reference to a Secret in the same namespace as the Provider, containing the authentication credentials for the provider API.

The Kubernetes secret can have any of the following keys:

address - overrides .spec.address
proxy - overrides .spec.proxy
token - used for authentication
username - overrides .spec.username
headers - HTTP headers values included in the POST request

Address example

For providers which embed tokens or other sensitive information in the URL, the incoming webhooks address can be stored in the secret using the address key:

---
apiVersion: v1
kind: Secret
metadata:
  name: my-provider-url
  namespace: default
stringData:
  address: "https://webhook.example.com/token"

Token example

For providers which require token based authentication, the API token can be stored in the secret using the token key:

---
apiVersion: v1
kind: Secret
metadata:
  name: my-provider-auth
  namespace: default
stringData:
  token: "my-api-token"

HTTP headers example

For providers which require specific HTTP headers to be attached to the POST request, the headers can be set in the secret using the headers key:

---
apiVersion: v1
kind: Secret
metadata:
  name: my-provider-headers
  namespace: default
stringData:
  headers: |
    Authorization: my-api-token
    X-Forwarded-Proto: https

Proxy auth example

Some networks need to use an authenticated proxy to access external services. Therefore, the proxy address can be stored as a secret to hide parameters like the username and password:

---
apiVersion: v1
kind: Secret
metadata:
  name: my-provider-proxy
  namespace: default
stringData:
  proxy: "http://username:password@proxy_url:proxy_port"

TLS certificates

.spec.certSecretRef is an optional field to specify a name reference to a Secret in the same namespace as the Provider, containing the TLS CA certificate. The secret must be of type kubernetes.io/tls or Opaque.

Example

To enable notification-controller to communicate with a provider API over HTTPS using a self-signed TLS certificate, set the ca.crt like so:

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: my-webhook
  namespace: flagger-system
spec:
  type: generic
  address: https://my-webhook.internal
  certSecretRef:
    name: my-ca-crt
---
apiVersion: v1
kind: Secret
metadata:
  name: my-ca-crt
  namespace: default
type: kubernetes.io/tls # or Opaque
stringData:
  ca.crt: |
    <--- CA Key --->

Warning: Support for the caFile key has been deprecated. If you have any Secrets using this key, the controller will log a deprecation warning.

HTTP/S proxy

.spec.proxy is an optional field to specify an HTTP/S proxy address.

If the proxy address contains sensitive information such as basic auth credentials, it is recommended to store the proxy in the Kubernetes secret referenced by .spec.secretRef.name. When the referenced Secret contains a proxy key, the .spec.proxy value is ignored.

Timeout

.spec.timeout is an optional field to specify the timeout for the HTTP/S request sent to the provider endpoint. The value must be in a Go recognized duration string format, e.g. 5m30s for a timeout of five minutes and thirty seconds.

Suspend

.spec.suspend is an optional field to suspend the provider. When set to true, the controller will stop sending events to this provider. When the field is set to false or removed, it will resume.

Working with Providers

Grafana

To send notifications to Grafana annotations API, enable the annotations on a Dashboard like so:

Annotations > Query > Enable Match any
Annotations > Query > Tags (Add Tag: flux)

If Grafana has authentication configured, create a Kubernetes Secret with the API token:

kubectl create secret generic grafana-token \
--from-literal=token=<grafana-api-key> \

Grafana can also use basic authorization to authenticate the requests, if both the token and the username/password are set in the secret, then token takes precedence over`basic auth:

kubectl create secret generic grafana-token \
--from-literal=username=<your-grafana-username> \
--from-literal=password=<your-grafana-password>

Create a provider of type grafana and reference the grafana-token secret:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: grafana
  namespace: default
spec:
  type: grafana
  address: https://<grafana-url>/api/annotations
  secretRef:
    name: grafana-token

GitHub dispatch

The githubdispatch provider generates GitHub events of type repository_dispatch for the selected repository. The repository_dispatch events can be used to trigger GitHub Actions workflow.

The request includes the event_type and client_payload fields:

event_type is generated from the involved object in the format {Kind}/{Name}.{Namespace}.
client_payload contains the Flux event.

Setting up the GitHub dispatch provider

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: github-dispatch
  namespace: flux-system
spec:
  type: githubdispatch
  address: https://github.com/stefanprodan/podinfo
  secretRef:
    name: api-token

The address is the address of your repository where you want to send webhooks to trigger GitHub workflows.

GitHub uses personal access tokens for authentication with its API:

GitHub personal access token

The provider requires a secret in the same format, with the personal access token as the value for the token key:

---
apiVersion: v1
kind: Secret
metadata:
  name: api-token
  namespace: default
stringData:
  token: <personal-access-tokens>

Setting up a GitHub workflow

To trigger a GitHub Actions workflow when a Flux Kustomization finishes reconciling, you need to set the event type for the repository_dispatch trigger to match the Flux object ID:

name: test-github-dispatch-provider
on:
  repository_dispatch:
    types: [Kustomization/podinfo.flux-system]

Assuming that we deploy all Flux kustomization resources in the same namespace, it will be useful to have a unique kustomization resource name for each application. This will allow you to use only event_type to trigger tests for the exact application.

Let's say we have following folder structure for applications kustomization manifests:

apps/
├── app1
│   └── overlays
│       ├── production
│       └── staging
└── app2
    └── overlays
        ├── production
        └── staging

You can then create a flux kustomization resource for the app to have unique event_type per app. The kustomization manifest for app1/staging:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta3
kind: Kustomization
metadata:
  name: app1
  namespace: flux-system
spec:
  path: "./app1/staging"

You would also like to know from the notification which cluster is being used for deployment. You can add the spec.summary field to the Flux alert configuration to mention the relevant cluster:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: github-dispatch
  namespace: flux-system
spec:
  summary: "staging (us-west-2)"
  providerRef:
    name: github-dispatch
  eventSeverity: info
  eventSources:
    - kind: Kustomization
      name: 'podinfo'

Now you can trigger the tests in the GitHub workflow for app1 in a staging cluster when the app1 resources defined in ./app1/staging/ are reconciled by Flux:

name: test-github-dispatch-provider
on:
  repository_dispatch:
    types: [Kustomization/podinfo.flux-system]
jobs:
  run-tests-staging:
    if: github.event.client_payload.metadata.summary == 'staging (us-west-2)'
    runs-on: ubuntu-18.04
    steps:
    - name: Run tests
      run: echo "running tests.."

Azure Event Hub

The Azure Event Hub supports two authentication methods, JWT and SAS based.

JWT based auth

In JWT we use 3 input values. Channel, token and address. We perform the following translation to match we the data we need to communicate with Azure Event Hub.

channel = Azure Event Hub namespace
address = Azure Event Hub name
token = JWT

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: azure
  namespace: default
spec:
  type: azureeventhub
  address: <event-hub-name>
  channel: <event-hub-namespace>
  secretRef:
    name: azure-token
---
apiVersion: v1
kind: Secret
metadata:
  name: azure-token
  namespace: default
stringData:
  token: <event-hub-token>

The controller doesn't take any responsibility for the JWT token to be updated. You need to use a secondary tool to make sure that the token in the secret is renewed.

If you want to make a easy test assuming that you have setup a Azure Enterprise application and you called it event-hub you can follow most of the bellow commands. You will need to provide the client_secret that you got when generating the Azure Enterprise Application.

export AZURE_CLIENT=$(az ad app list --filter "startswith(displayName,'event-hub')" --query '[].appId' |jq -r '.[0]')
export AZURE_SECRET='secret-client-secret-generated-at-creation'
export AZURE_TENANT=$(az account show -o tsv --query tenantId)

curl -X GET --data 'grant_type=client_credentials' --data "client_id=$AZURE_CLIENT" --data "client_secret=$AZURE_SECRET" --data 'resource=https://eventhubs.azure.net' -H 'Content-Type: application/x-www-form-urlencoded' https://login.microsoftonline.com/$AZURE_TENANT/oauth2/token |jq .access_token

Use the output you got from curl and add it to your secret like bellow:

kubectl create secret generic azure-token \
--from-literal=token='A-valid-JWT-token'

SAS based auth

When using SAS auth, we only use the address field in the secret.

---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: azure
  namespace: default
spec:
  type: azureeventhub
  secretRef:
    name: azure-webhook
---
apiVersion: v1
kind: Secret
metadata:
  name: azure-webhook
  namespace: default
stringData:
  address: <SAS-URL>

Assuming that you have created the Azure event hub and namespace you should be able to use a similar command to get your connection string. This will give you the default Root SAS, which is NOT supposed to be used in production.

az eventhubs namespace authorization-rule keys list --resource-group <rg-name> --namespace-name <namespace-name> --name RootManageSharedAccessKey -o tsv --query primaryConnectionString
# The output should look something like this:
Endpoint=sb://fluxv2.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=yoursaskeygeneatedbyazure

To create the needed secret:

kubectl create secret generic azure-webhook \
--from-literal=address="Endpoint=sb://fluxv2.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=yoursaskeygeneatedbyazure"

Git Commit Status Updates

The notification-controller can mark Git commits as reconciled by posting Flux Kustomization events to the origin repository using Git SaaS providers APIs.

Example

The following is an example of how to update the Git commit status for the GitHub repository where Flux was bootstrapped with flux bootstrap github --owner=my-gh-org --repository=my-gh-repo.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: github-status
  namespace: flux-system
spec:
  type: github
  address: https://github.com/my-gh-org/my-gh-repo
  secretRef:
    name: github-token
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: github-status
  namespace: flux-system
spec:
  providerRef:
    name: github-status
  eventSources:
    - kind: Kustomization
      name: flux-system

GitHub

When .spec.type is set to github, the referenced secret must contain a key called token with the value set to a GitHub personal access token.

The token must have permissions to update the commit status for the GitHub repository specified in .spec.address.

You can create the secret with kubectl like this:

kubectl create secret generic github-token --from-literal=token=<GITHUB-TOKEN>

GitLab

When .spec.type is set to gitlab, the referenced secret must contain a key called token with the value set to a GitLab personal access token.

The token must have permissions to update the commit status for the GitLab repository specified in .spec.address.

You can create the secret with kubectl like this:

kubectl create secret generic gitlab-token --from-literal=token=<GITLAB-TOKEN>

Gitea

When .spec.type is set to gitea, the referenced secret must contain a key called token with the value set to a Gitea token.

The token must have at least the write:repository permission for the provider to update the commit status for the Gitea repository specified in .spec.address.

{{% alert color="info" title="Gitea 1.20.0 & 1.20.1" %}} Due to a bug in Gitea 1.20.0 and 1.20.1, these versions require the additional read:misc scope to be applied to the token. {{% /alert %}}

You can create the secret with kubectl like this:

kubectl create secret generic gitea-token --from-literal=token=<GITEA-TOKEN>

BitBucket

When .spec.type is set to bitbucket, the referenced secret must contain a key called token with the value set to a BitBucket username and an app password in the format <username>:<app-password>.

The app password must have Repositories (Read/Write) permission for the BitBucket repository specified in .spec.address.

You can create the secret with kubectl like this:

kubectl create secret generic bitbucket-token --from-literal=token=<username>:<app-password>

BitBucket Server/Data Center

When .spec.type is set to bitbucketserver, the following auth methods are available:

Basic Authentication (username/password)
HTTP access tokens

For Basic Authentication, the referenced secret must contain a password field. The username field can either come from the .spec.username field of the Provider or can be defined in the referenced secret.

You can create the secret with kubectl like this:

kubectl create secret generic bb-server-username-password --from-literal=username=<username> --from-literal=password=<password>

For HTTP access tokens, the secret can be created with kubectl like this:

kubectl create secret generic bb-server-token --from-literal=token=<token>

The HTTP access token must have Repositories (Read/Write) permission for the repository specified in .spec.address.

NOTE: Please provide HTTPS clone URL in the address field of this provider. SSH URLs are not supported by this provider type.

Azure DevOps

When .spec.type is set to azuredevops, the referenced secret must contain a key called token with the value set to a Azure DevOps personal access token.

The token must have permissions to update the commit status for the Azure DevOps repository specified in .spec.address.

You can create the secret with kubectl like this:

kubectl create secret generic azuredevops-token --from-literal=token=<AZURE-TOKEN>

Files

providers.md

Latest commit

History

providers.md

File metadata and controls

Providers

Example

Writing a provider spec

Type

Alerting

Generic webhook

Generic webhook with HMAC

HMAC verification example

Slack

Slack example

Slack (legacy) example

Microsoft Teams

Microsoft Teams example

DataDog

DataDog example

Discord

Discord example

Sentry

Sentry example

Telegram

Telegram example

Matrix

Matrix example

Lark

Lark example

Rocket

Rocket example

Google Chat

Google Chat example

Google Pub/Sub

Google Pub/Sub with JSON Credentials and Custom Headers Example

Opsgenie

Opsgenie example

PagerDuty

PagerDuty example

Prometheus Alertmanager

Prometheus Alertmanager example

Webex

Webex example

NATS

NATS with Username/Password Credentials Example

Address

Channel

Username

Secret reference

Address example

Token example

HTTP headers example

Proxy auth example

TLS certificates

Example

HTTP/S proxy

Timeout

Suspend

Working with Providers

Grafana

GitHub dispatch

Setting up the GitHub dispatch provider

Setting up a GitHub workflow

Azure Event Hub

JWT based auth

SAS based auth

Git Commit Status Updates

Example

GitHub

GitLab

Gitea

BitBucket

BitBucket Server/Data Center

Azure DevOps