ResourceSet dynamic inputs from OCI Repository Tags

[stefanprodan]

With the introduction of ResourceSet input providers, Flux Operator can dynamically deploy applications based on data provided by Git services e.g. Pull Requests or Git branches.

There are cases when the Kubernetes manifests are pushed from Git to a container registry and stored as OCI artifacts. One such example is a mono-repo with multiple microservices, where changes to a specific microservice, trigger a new OCI artifact push and Flux running in the cluster should deploy only the affected microservice.

In this scenario, the OCI repository tags could be used as inputs for the ResourceSet templates and thus decouple the deployment process from Git. To support this use case, a new ResourceSetInputProvider type will be introduced called OCIRepositoryTag.

Example

We assume that the Kubernetes manifests of all microservices are stored in the same OCI repository. When a commit is pushed to the Git mono-repo, the CI pipeline determines which microservice has changed and pushes the Kubernetes manifests belonging to the that microservice using a tag that matches the format app-<microservice name>.

flux push artifact oci://ghcr.io/<org>/<mono-repo>:app-microservice1 --path ./microservice1

Flux Operator Config

We define a ResourceSetInputProvider that fetches the OCI repository tags from the specified URL and filters them by a regular expression:

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
  name: app-microservices-tags
  namespace: default
  annotations:
    fluxcd.controlplane.io/reconcile: "enabled"
    fluxcd.controlplane.io/reconcileEvery: "5m"
spec:
    type: OCIRepositoryTag
    url: oci://ghcr.io/<org>/<mono-repo>
    secretRef:
      name: ghcr-pull-secret
    filter:
      includeTag: "^app-.*"

Assuming we have three microservices in the mono-repo, the ResourceSetInputProvider will export the following inputs:

status:
  exportedInputs:
    - id: "1433470881"
      digest: "sha256:d7e924b4882e55b97627355c7b3d2e711e9b54303afa2f50c25377f4df66a83b"
      tag: 'app-microservice1'
    - id: "1433536418"
      digest: "sha256:3b6cdcc7adcc9a84d3214ee1c029543789d90b5ae69debe9efa3f66e982875de"
      tag: 'app-microservice2'
    - id: "1433601955"
      digest: "sha256:a9cb966cddc1a0c56dc0d57dda485d9477dd397f8b45f222717b24663471fd1f"
      tag: 'app-microservice3'

To speed up the registry scan, a Flux Receiver could be called from the CI job that pushes the OCI artifacts:

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: registry-scan-webhook
  namespace: apps
spec:
  type: generic-hmac
  secretRef:
    name: receiver-token
  resources:
    - apiVersion: fluxcd.controlplane.io/v1
      kind: ResourceSetInputProvider
      name: app-microservices-tags

To deploy the microservices, we'll create a ResourceSet that uses the inputs from the ResourceSetInputProvider:

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
  name: app-microservices
  namespace: apps
spec:
  inputsFrom:
    - kind: ResourceSetInputProvider
      name: app-microservices-tags
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1beta2
      kind: OCIRepository
      metadata:
        name: << inputs.tag | quote >>
        namespace: apps
      spec:
        interval: 1h
        url: oci://ghcr.io/<org>/<mono-repo>
        ref:
          digest: << inputs.digest | quote >>
        secretRef:
          name: ghcr-pull-secret
    - apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: << inputs.tag | quote >>
        namespace: apps
      spec:
        interval: 1h
        retryInterval: 5m
        timeout: 5m
        sourceRef:
          kind: OCIRepository
          name: << inputs.tag | quote >>
        path: overlays/production
        prune: true
        wait: true

Every time a microservice artifact is pushed to the repository, the ResourceSetInputProvider reconciler will update its digest and Flux will upgrade the microservice in the cluster using the manifests pulled from the registry by digest.

If a new microservice is added to the mono-repo, the exported inputs will include the new tag and the new microservice will be created by Flux.

If a microservice is removed from the mono-repo and its tag is deleted from the registry, Flux will uninstall the microservice from the cluster. Another option to trigger the removal of a microservice is to add its name to the excludeTag filter of the ResourceSetInputProvider.