How expose Supervisor via Cloudflare #1219

junoriosity · 2022-07-03T21:39:07Z

junoriosity
Jul 3, 2022

I am currently trying to make Supervisor work. I first installed it via Terraform and Helm

resource "helm_release" "pinniped" {
  name = "pinniped"
  namespace = "pinniped-supervisor"
  create_namespace = true
  repository = "https://charts.bitnami.com/bitnami"
  chart = "pinniped"

  version = "0.1.5"

  values = [
    "${file("${path.module}/yaml_files/pinniped/supervisor_values.yaml")}"
  ]
}

where the supervisor_values.yaml are

supervisor:
  enabled: true

  service:
    type: LoadBalancer

concierge:
  enabled: false

I get an IP address from the load balancer and put it into an A record at Cloudflare

Usually, I get the TLS from Cloudflare automatically and also here that seems to be the case. However, when I try to access supervisor.my-domain.com I get

I also tried to follow this tutorial here with the adaptions for Cloudflare.

I even see the secret coming up via

kubectl get secret supervisor-tls-cert --namespace pinniped-supervisor

but when I call supervisor.my-domain.com again, there is still the same result.

Answered by cfryanr

Aug 3, 2022

Happy to help, @junoriosity. I'm curious, did you figure out what was the cause of the problem? It might help others who read this discussion if you could describe your solution.

As for marking the comment as the answer, it appears that it cannot be done inside a threaded discussion, according to the GitHub docs.

View full answer

cfryanr · 2022-07-06T16:33:00Z

cfryanr
Jul 6, 2022
Maintainer

Hi @junoriosity,

When you created your FederationDomain, did you configure it to use that Secret as its TLS secret? See FederationDomain API docs and the section called "Configuring TLS for the Supervisor OIDC endpoints" in this doc for more details.

Also check the Pod logs of the Supervisor Pods to see if they are complaining about loading that TLS secret.

0 replies

junoriosity · 2022-07-17T12:36:38Z

junoriosity
Jul 17, 2022
Author

Hi @cfryanr,

many thanks for getting back to me. I tried to mimick the exactly same behavior and yet I got a new issue. Here, is the workflow.

I installed the supervisor like this

kubectl apply -f https://get.pinniped.dev/v0.18.0/install-pinniped-supervisor.yaml

and the load balancer like this

apiVersion: v1
kind: Service
metadata:
  name: pinniped-supervisor-loadbalancer
  namespace: pinniped-supervisor
spec:
  type: LoadBalancer
  selector:
    app: pinniped-supervisor
  ports:
  - protocol: TCP
    port: 443
    targetPort: 8443 # 8443 is the TLS port.

Then I point in Cloudflare supervisor.my-domain.com to the LoadBalancer`s IP.

Following that, I run in Terraform the following code:

resource "helm_release" "cert-manager" {
  count = var.is_main_cluster ? 1 : 0
  name = "cm"
  namespace = "cert-manager"
  create_namespace = true
  repository = "https://charts.jetstack.io"
  chart = "cert-manager"

  version = "1.8.2"

  set {
    name = "installCRDs"
    value = true
  }
}

resource "kubectl_manifest" "cloudflare-api-token-secret" {
  count = var.is_main_cluster ? 1 : 0
    yaml_body = <<YAML
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: pinniped-supervisor
type: Opaque
stringData:
  api-token: ${var.cloudflare_api_token}
YAML
}

resource "kubectl_manifest" "pinniped-issuer" {
  count = var.is_main_cluster ? 1 : 0
    yaml_body = <<YAML
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: pinniped-issuer
  namespace: pinniped-supervisor
spec:
  acme:
    email: <my-cloudflare-email>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: pinniped-issuer
    solvers:
    - dns01:
        cloudflare:
          email: <my-cloudflare-email>
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token
YAML
}

resource "kubectl_manifest" "pinniped-cert" {
  count = var.is_main_cluster ? 1 : 0
    yaml_body = <<YAML
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: pinniped-certificate
  namespace: pinniped-supervisor
spec:
  secretName: supervisor-tls-cert
  issuerRef:
    name: pinniped-issuer

  duration: 2160h # 90d
  renewBefore: 720h # 30d before SSL will expire, renew it
  dnsNames:
    - "supervisor.my-domain.com"
YAML
}

resource "kubectl_manifest" "fedeeration-domain" {
  count = var.is_main_cluster ? 1 : 0
    yaml_body = <<YAML
apiVersion: config.supervisor.pinniped.dev/v1alpha1
kind: FederationDomain
metadata:
  name: demo-federation-domain
  namespace: pinniped-supervisor
spec:
  # You can choose an arbitrary path for the issuer URL.
  issuer: "https://supervisor.my-domain.com/demo-issuer"
  tls:
    secretName: supervisor-tls-cert
YAML
}

When I now try to reach supervisor.my-domain.com I get 404 page not found.

Here are the logs of the supervisor pod:

{"level":"error","timestamp":"2022-07-17T12:11:53.915485Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"FederationDomainWatcherController: { } failed with: could not update status: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:53.917604Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-oidc-provider-upstream-state-signature-key-controller: {pinniped-supervisor demo-federation-domain} failed with: failed to create or update secret: failed to create secret pinniped-supervisor/pinniped-oidc-provider-upstream-state-signature-key-332d6e1e-a5c0-47c3-8e6e-6329b8530d58: Post \"https://192.168.64.1:443/api/v1/namespaces/pinniped-supervisor/secrets\": middleware request for &kubeclient.request{verb:\"create\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"\", Version:\"v1\", Resource:\"secrets\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000), (func(kubeclient.Object) error)(0x1a37fe0)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:53.918378Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"JWKSController: {pinniped-supervisor demo-federation-domain} failed with: cannot create or update secret: cannot create secret: Post \"https://192.168.64.1:443/api/v1/namespaces/pinniped-supervisor/secrets\": middleware request for &kubeclient.request{verb:\"create\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"\", Version:\"v1\", Resource:\"secrets\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000), (func(kubeclient.Object) error)(0x1a37fe0)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:53.918960Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-oidc-provider-upstream-state-encryption-key-controller: {pinniped-supervisor demo-federation-domain} failed with: failed to create or update secret: failed to create secret pinniped-supervisor/pinniped-oidc-provider-upstream-state-encryption-key-332d6e1e-a5c0-47c3-8e6e-6329b8530d58: Post \"https://192.168.64.1:443/api/v1/namespaces/pinniped-supervisor/secrets\": middleware request for &kubeclient.request{verb:\"create\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"\", Version:\"v1\", Resource:\"secrets\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000), (func(kubeclient.Object) error)(0x1a37fe0)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:53.919519Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-oidc-provider-hmac-key-controller: {pinniped-supervisor demo-federation-domain} failed with: failed to create or update secret: failed to create secret pinniped-supervisor/pinniped-oidc-provider-hmac-key-332d6e1e-a5c0-47c3-8e6e-6329b8530d58: Post \"https://192.168.64.1:443/api/v1/namespaces/pinniped-supervisor/secrets\": middleware request for &kubeclient.request{verb:\"create\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"\", Version:\"v1\", Resource:\"secrets\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000), (func(kubeclient.Object) error)(0x1a37fe0)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:53.927814Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"FederationDomainWatcherController: { } failed with: could not update status: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:53.945085Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"FederationDomainWatcherController: { } failed with: could not update status: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:53.971629Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"FederationDomainWatcherController: { } failed with: could not update status: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:54.018476Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"FederationDomainWatcherController: { } failed with: could not update status: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:54.302553Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"FederationDomainWatcherController: { } failed with: could not update status: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:54.502299Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-oidc-provider-upstream-state-signature-key-controller: {pinniped-supervisor demo-federation-domain} failed with: failed to create or update secret: failed to create secret pinniped-supervisor/pinniped-oidc-provider-upstream-state-signature-key-332d6e1e-a5c0-47c3-8e6e-6329b8530d58: Post \"https://192.168.64.1:443/api/v1/namespaces/pinniped-supervisor/secrets\": middleware request for &kubeclient.request{verb:\"create\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"\", Version:\"v1\", Resource:\"secrets\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000), (func(kubeclient.Object) error)(0x1a37fe0)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:54.702707Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"JWKSController: {pinniped-supervisor demo-federation-domain} failed with: cannot create or update secret: cannot create secret: Post \"https://192.168.64.1:443/api/v1/namespaces/pinniped-supervisor/secrets\": middleware request for &kubeclient.request{verb:\"create\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"\", Version:\"v1\", Resource:\"secrets\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000), (func(kubeclient.Object) error)(0x1a37fe0)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:54.902330Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-oidc-provider-hmac-key-controller: {pinniped-supervisor demo-federation-domain} failed with: failed to create or update secret: failed to create secret pinniped-supervisor/pinniped-oidc-provider-hmac-key-332d6e1e-a5c0-47c3-8e6e-6329b8530d58: Post \"https://192.168.64.1:443/api/v1/namespaces/pinniped-supervisor/secrets\": middleware request for &kubeclient.request{verb:\"create\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"\", Version:\"v1\", Resource:\"secrets\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000), (func(kubeclient.Object) error)(0x1a37fe0)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:55.102426Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-oidc-provider-upstream-state-signature-key-controller: {pinniped-supervisor demo-federation-domain} failed with: failed to update federationdomain: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:55.102673Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-oidc-provider-upstream-state-encryption-key-controller: {pinniped-supervisor demo-federation-domain} failed with: failed to create or update secret: failed to create secret pinniped-supervisor/pinniped-oidc-provider-upstream-state-encryption-key-332d6e1e-a5c0-47c3-8e6e-6329b8530d58: Post \"https://192.168.64.1:443/api/v1/namespaces/pinniped-supervisor/secrets\": middleware request for &kubeclient.request{verb:\"create\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"\", Version:\"v1\", Resource:\"secrets\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000), (func(kubeclient.Object) error)(0x1a37fe0)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:56.302800Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-oidc-provider-upstream-state-signature-key-controller: {pinniped-supervisor demo-federation-domain} failed with: failed to update federationdomain: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:56.502451Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-oidc-provider-upstream-state-encryption-key-controller: {pinniped-supervisor demo-federation-domain} failed with: failed to update federationdomain: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:56.902867Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"JWKSController: {pinniped-supervisor demo-federation-domain} failed with: cannot update FederationDomain: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:58.103062Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-oidc-provider-upstream-state-encryption-key-controller: {pinniped-supervisor demo-federation-domain} failed with: failed to update federationdomain: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-17T12:11:58.703320Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"JWKSController: {pinniped-supervisor demo-federation-domain} failed with: cannot update FederationDomain: Put \"https://192.168.64.1:443/apis/config.supervisor.pinniped.dev/v1alpha1/namespaces/pinniped-supervisor/federationdomains/demo-federation-domain/status\": middleware request for &kubeclient.request{verb:\"update\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"config.supervisor.pinniped.dev\", Version:\"v1alpha1\", Resource:\"federationdomains\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aad000)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"status\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"info","timestamp":"2022-07-17T12:28:52.531575Z","caller":"log/log.go:195$log.(*Logger).Output","message":"http: TLS handshake error from 10.0.0.3:12943: EOF\n"}

Do you have an idea what I did wrong?

6 replies

cfryanr Jul 27, 2022
Maintainer

Hi @junoriosity,

Sorry for the slow response.

If you have more than one Supervisor pod running, then it is normal for all of the pods except for one to have errors that say write attempt rejected as client is not leader. One of the pods will be automatically elected to be the leader, and only that pod can perform writes. The follower pods will complain about not performing writes, but they are designed to not need to be able to perform writes, so you can ignore those log messages, as long as your leader pod is working okay.

So in this case, you might have looked at the wrong supervisor pod. If you want to definitively check which Supervisor pod is the current leader, then you can look inside the Lease resource in the Supervisor's namespace, or you can see the log message in the leader's pod logs where it says that it was elected as the leader.

As for the 404 not found error, there is nothing listening for requests at the root path of the server. So, for example, curl https://supervisor.my-domain.com should be a 404, even with a fully configured Supervisor. You configured the issuer path to be https://supervisor.my-domain.com/demo-issuer, so if everything is working correctly then you should be able to curl https://supervisor.my-domain.com/demo-issuer/.well-known/openid-configuration, which is your FederationDomain's OIDC discovery endpoint. If everything is working, then that endpoint should return 200 OK with some JSON in the response body. Note that if the CA which signed your Supervisor's TLS cert is not trusted by you client machine, then you'll also need to use the curl --cacert flag to inform curl of your CA's public cert.

junoriosity Jul 29, 2022
Author

Hi @cfryanr

many thanks for your reply. I tried now some new things. These are my values for Pinniped supervisor

fullnameOverride: pinniped-supervisor

supervisor:
  enabled: true

  service:
    type: ClusterIP

concierge:
  enabled: false

which I install in Terraform with

resource "helm_release" "pinniped" {
  count = var.is_main_cluster ? 1 : 0
  name = "pinniped"
  namespace = "pinniped-supervisor"
  create_namespace = true
  repository = "https://charts.bitnami.com/bitnami"
  chart = "pinniped"

  version = "0.1.5"

  values = [
    "${file("${path.module}/yaml_files/pinniped/supervisor_values.yaml")}"
  ]
}

In addition, I create TLS certificates in Cloudflare like here and install a tls secret, ingress, and federation domain like this

resource "kubernetes_secret" "cloudflare-secret" {
  count = var.is_main_cluster ? 1 : 0

  depends_on = [ helm_release.pinniped ]

  metadata {
    name = "cloudflare-tls"
    namespace = "pinniped-supervisor"
  }

  data = {
    "tls.crt" = file("${var.cloudflare_cert_path}/origin-ca.crt")
    "tls.key" = file("${var.cloudflare_cert_path}/origin-ca.pk")
  }

  type = "kubernetes.io/tls"
}

resource "kubectl_manifest" "supervisor-service" {
  count = var.is_main_cluster ? 1 : 0

  depends_on = [ helm_release.pinniped ]

    yaml_body = <<YAML
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pinniped-supervisor-supervisor
  namespace: pinniped-supervisor
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
    - hosts:
        - supervisor.my-domain.com
      secretName: cloudflare-tls
  rules:
    - host: supervisor.my-domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: pinniped-supervisor-supervisor
                port:
                  number: 443
  ingressClassName: nginx
YAML
}

resource "kubectl_manifest" "fedeeration-domain" {
  count = var.is_main_cluster ? 1 : 0
    yaml_body = <<YAML
apiVersion: config.supervisor.pinniped.dev/v1alpha1
kind: FederationDomain
metadata:
  name: demo-federation-domain
  namespace: pinniped-supervisor
spec:
  # You can choose an arbitrary path for the issuer URL.
  issuer: "https://supervisor.my-domain.com/demo-issuer"
  tls:
    secretName: cloudflare-tls
YAML
}

I can install everything and it works nicely. However, I get the same issue again with invalid TLS certs

and

curl https://supervisor.my-domain.com/demo-issuer/.well-known/openid-configuration
pinniped supervisor has invalid TLS serving certificate configuration

curl https://supervisor.my-domain.com/demo-issuer/.well-known/openid-configuration --cacert ~/origin-ca.crt
pinniped supervisor has invalid TLS serving certificate configuration

The strange thing is that I set in Cloudflare everything to strict mode, which just allows traffic with valid certs

And all other ingresses are blocked now apart from everything starting with https://supervisor.my-domain.com. Hence, these certs must have some impact, only that I still keep getting the same error.

These are the logs of the supervisor pod:

{"level":"info","timestamp":"2022-07-29T14:39:56.273833Z","caller":"go.pinniped.dev/internal/supervisor/server/server.go:507$server.main","message":"Running supervisor","user-agent":"pinniped-supervisor/v0.0.0 (linux/amd64) kubernetes/ec533cd","version":{"major":"0","minor":"0","gitVersion":"v0.0.0","gitCommit":"ec533cd781d390130887c06aaa44cb2e44f76b57","gitTreeState":"clean","buildDate":"2022-06-10T10:03:00Z","goVersion":"go1.18.1","compiler":"gc","platform":"linux/amd64"},"arguments":["pinniped-supervisor","/etc/podinfo","/bitnami/pinniped/conf/pinniped.yaml"]}
{"level":"error","timestamp":"2022-07-29T14:39:56.612639Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-supervisor-supervisor-secret-generator: {pinniped-supervisor pinniped-supervisor-supervisor-key} failed with: failed to create/update secret pinniped-supervisor/pinniped-supervisor-supervisor-key: Post \"https://192.168.64.1:443/api/v1/namespaces/pinniped-supervisor/secrets\": middleware request for &kubeclient.request{verb:\"create\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"\", Version:\"v1\", Resource:\"secrets\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aaccc0), (func(kubeclient.Object) error)(0x1a37ca0)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"info","timestamp":"2022-07-29T14:39:56.614477Z","caller":"k8s.io/[email protected]/tools/leaderelection/leaderelection.go:248$leaderelection.(*LeaderElector).acquire","message":"attempting to acquire leader lease pinniped-supervisor/pinniped-supervisor-supervisor...\n"}
{"level":"error","timestamp":"2022-07-29T14:39:56.618967Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-supervisor-supervisor-secret-generator: {pinniped-supervisor pinniped-supervisor-supervisor-key} failed with: failed to create/update secret pinniped-supervisor/pinniped-supervisor-supervisor-key: Post \"https://192.168.64.1:443/api/v1/namespaces/pinniped-supervisor/secrets\": middleware request for &kubeclient.request{verb:\"create\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"\", Version:\"v1\", Resource:\"secrets\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aaccc0), (func(kubeclient.Object) error)(0x1a37ca0)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"error","timestamp":"2022-07-29T14:39:56.630412Z","caller":"go.pinniped.dev/internal/controllerlib/controller.go:219$controllerlib.(*controller).handleKey","message":"pinniped-supervisor-supervisor-secret-generator: {pinniped-supervisor pinniped-supervisor-supervisor-key} failed with: failed to create/update secret pinniped-supervisor/pinniped-supervisor-supervisor-key: Post \"https://192.168.64.1:443/api/v1/namespaces/pinniped-supervisor/secrets\": middleware request for &kubeclient.request{verb:\"create\", namespace:\"pinniped-supervisor\", resource:schema.GroupVersionResource{Group:\"\", Version:\"v1\", Resource:\"secrets\"}, reqFuncs:[]func(kubeclient.Object) error{(func(kubeclient.Object) error)(0x1aaccc0), (func(kubeclient.Object) error)(0x1a37ca0)}, respFuncs:[]func(kubeclient.Object) error(nil), subresource:\"\"} failed to mutate: request mutation failed: write attempt rejected as client is not leader\n"}
{"level":"info","timestamp":"2022-07-29T14:39:56.632437Z","caller":"k8s.io/[email protected]/tools/leaderelection/leaderelection.go:258$leaderelection.(*LeaderElector).acquire.func1","message":"successfully acquired lease pinniped-supervisor/pinniped-supervisor-supervisor\n"}
{"level":"info","timestamp":"2022-07-29T14:44:11.924705Z","caller":"log/log.go:195$log.(*Logger).Output","message":"http: TLS handshake error from 127.0.0.1:34030: remote error: tls: unknown certificate\n"}
{"level":"info","timestamp":"2022-07-29T14:44:12.018681Z","caller":"log/log.go:195$log.(*Logger).Output","message":"http: TLS handshake error from 127.0.0.1:34032: remote error: tls: unknown certificate\n"}
{"level":"info","timestamp":"2022-07-29T14:44:12.523442Z","caller":"log/log.go:195$log.(*Logger).Output","message":"http: TLS handshake error from 127.0.0.1:34028: EOF\n"}
{"level":"info","timestamp":"2022-07-29T14:44:23.956510Z","caller":"log/log.go:195$log.(*Logger).Output","message":"http: TLS handshake error from 127.0.0.1:34104: remote error: tls: unknown certificate\n"}
{"level":"info","timestamp":"2022-07-29T14:44:23.958590Z","caller":"log/log.go:195$log.(*Logger).Output","message":"http: TLS handshake error from 127.0.0.1:34106: remote error: tls: unknown certificate\n"}
{"level":"info","timestamp":"2022-07-29T15:57:54.792941Z","caller":"log/log.go:195$log.(*Logger).Output","message":"http: TLS handshake error from 127.0.0.1:60480: EOF\n"}

Do you have an idea how I can resolve the TLS issue?

cfryanr Aug 2, 2022
Maintainer

Hi @junoriosity,

The message pinniped supervisor has invalid TLS serving certificate configuration means that it is not finding a valid TLS cert configured for the FederationDomain's hostname. Does your cloudflare-tls Secret exist, is it in the same namespace where the Supervisor is installed, and does its contents look correct?

As first step, try increasing the Supervisor's log level to debug so you can get more debugging log messages into the pod logs. Then, see if you get any of the debug log messages that can be output by this controller (the plog.Debug lines of code), or any other messages related to loading TLS certs: https://github.com/vmware-tanzu/pinniped/blob/main/internal/controller/supervisorconfig/tls_cert_observer.go. This is the controller which loads the TLS certs for the FederationDomains from Secrets.

If that doesn't provide any hints, then turn the log level up to trace and look for this log message after trying to curl the well-known endpoint.

pinniped/internal/supervisor/server/server.go

Lines 430 to 436 in a876591

    
           plog.Trace("GetCertificate called", 
        
           	"info.ServerName", info.ServerName, 
        
           	"foundSNICert", cert != nil, 
        
           	"foundDefaultCert", defaultCert != nil, 
        
           	"host", host, 
        
           	"port", port, 
        
           )

This log message will appear on every incoming request to give some hints regarding which TLS cert is being used to handle the request.

Hopefully you can get some hints from these logs.

junoriosity Aug 3, 2022
Author

Hi @cfryanr,

many thanks for your input. That has fixed it for me. You are awesome. :)

junoriosity Aug 3, 2022
Author

@cfryanr How can I mark your last reply as an answer? I cannot find it, but it is so well-deserved :)

cfryanr · 2022-08-03T19:26:25Z

cfryanr
Aug 3, 2022
Maintainer

Happy to help, @junoriosity. I'm curious, did you figure out what was the cause of the problem? It might help others who read this discussion if you could describe your solution.

As for marking the comment as the answer, it appears that it cannot be done inside a threaded discussion, according to the GitHub docs.

1 reply

junoriosity Aug 3, 2022
Author

Yeah sure, the reason was simple, the cloudflare-tls-secret was just in the wrong namespace. But it now works nicely.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How expose Supervisor via Cloudflare #1219

{{title}}

Replies: 3 comments 7 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How expose Supervisor via Cloudflare #1219

junoriosity Jul 3, 2022

Replies: 3 comments · 7 replies

cfryanr Jul 6, 2022 Maintainer

junoriosity Jul 17, 2022 Author

cfryanr Jul 27, 2022 Maintainer

junoriosity Jul 29, 2022 Author

cfryanr Aug 2, 2022 Maintainer

junoriosity Aug 3, 2022 Author

junoriosity Aug 3, 2022 Author

cfryanr Aug 3, 2022 Maintainer

junoriosity Aug 3, 2022 Author

junoriosity
Jul 3, 2022

Replies: 3 comments 7 replies

cfryanr
Jul 6, 2022
Maintainer

junoriosity
Jul 17, 2022
Author

cfryanr Jul 27, 2022
Maintainer

junoriosity Jul 29, 2022
Author

cfryanr Aug 2, 2022
Maintainer

junoriosity Aug 3, 2022
Author

junoriosity Aug 3, 2022
Author

cfryanr
Aug 3, 2022
Maintainer

junoriosity Aug 3, 2022
Author