Issue in using kubeapps with Pinniped

[junoriosity]

I want to make Kubeapps work with Pinniped authentication.

For this, I first install Pinniper conierge and supervisor like this, although I do some modifications to the certs by using Cloudflare (the following is written in Terraform code, but the YAML files should be readily seen and whenever something appears like count = var.is_main_cluster ? 1 : 0, this is only means that this resource is only deployed to the main cluster):

resource "kubernetes_secret" "cloudflare-secret" {
  count = var.is_main_cluster ? 1 : 0

  depends_on = [ kubectl_manifest.pinniped_supervisor ]

  metadata {
    name = "cloudflare-tls"
    namespace = "pinniped-supervisor"
  }

  data = {
    "tls.crt" = file("${var.cloudflare_cert_path}/origin-ca.crt")
    "tls.key" = file("${var.cloudflare_cert_path}/origin-ca.pk")
  }

  type = "kubernetes.io/tls"
}

resource "kubectl_manifest" "federation-domain" {
  count = var.is_main_cluster ? 1 : 0
    yaml_body = <<YAML
apiVersion: config.supervisor.pinniped.dev/v1alpha1
kind: FederationDomain
metadata:
  name: demo-federation-domain
  namespace: pinniped-supervisor
spec:
  # You can choose an arbitrary path for the issuer URL.
  issuer: "https://supervisor.my-domain.com/issuer"
  tls:
    secretName: cloudflare-tls
YAML
}


resource "kubectl_manifest" "auth0_client_credentials" {
  count = var.is_main_cluster ? 1 : 0
    yaml_body = <<YAML
apiVersion: v1
kind: Secret
metadata:
  namespace: pinniped-supervisor
  name: auth0-client-credentials
type: secrets.pinniped.dev/oidc-client
stringData:
  # This should be the app's "Client ID"
  clientID: ${var.auth0_client_id}
  # This should be the app's "Client secret"
  clientSecret: ${var.auth0_client_secret}
YAML
}

resource "kubectl_manifest" "oidc_provider" {
  depends_on = [ kubectl_manifest.auth0_client_credentials ]

  count = var.is_main_cluster ? 1 : 0
    yaml_body = <<YAML
apiVersion: idp.supervisor.pinniped.dev/v1alpha1
kind: OIDCIdentityProvider
metadata:
  namespace: pinniped-supervisor
  name: auth0
spec:
  # This should be the app's "Auth0 Domain".
  issuer: ${var.auth0_domain}/
  authorizationConfig:
    additionalScopes: [groups, email, offline_access]
  claims:
    username: email
    groups: groups
  client:
    secretName: auth0-client-credentials
YAML
}


data "kubectl_file_documents" "pinniped_concierge_crds" {
    content = file("${path.module}/yaml_files/pinniped/install-pinniped-concierge-crds.yaml")
}

resource "kubectl_manifest" "pinniped_concierge_crds" {
    count = length(data.kubectl_file_documents.pinniped_concierge_crds.documents)
    yaml_body = element(data.kubectl_file_documents.pinniped_concierge_crds.documents, count.index)
}

data "kubectl_file_documents" "pinniped_concierge_resources" {
    content = file("${path.module}/yaml_files/pinniped/install-pinniped-concierge-resources.yaml")
}

resource "kubectl_manifest" "pinniped_concierge_resources" {
    depends_on = [ kubectl_manifest.pinniped_concierge_crds ]
    count = length(data.kubectl_file_documents.pinniped_concierge_resources.documents)
    yaml_body = element(data.kubectl_file_documents.pinniped_concierge_resources.documents, count.index)
}

resource "kubectl_manifest" "jwt_authenticator" {
  depends_on = [ kubectl_manifest.pinniped_concierge_resources ]
  
  yaml_body = <<YAML
apiVersion: authentication.concierge.pinniped.dev/v1alpha1
kind: JWTAuthenticator
metadata:
  name: supervisor-jwt-authenticator
spec:
  # This should be the issuer URL that was declared in the FederationDomain.
  issuer: "https://supervisor.my-domain.com/issuer"
  # Replace this with your own unique value.
  audience: ${var.context}-${random_password.audience_append.result}
YAML
}

Now, this works like a charm. But the next thing is to install kubeapps with the helm charts and the following values:

useHelm3: true
ingress:
  enabled: true
  hostname: kubeapps.my-domain.com
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"

authProxy:
  extraFlags:
    - --oidc-issuer-url=https://my-domain.eu.auth0.com/ # Example: https://antoniogamez.eu.auth0.com/  # mind the last "/" character
  clientID: my-client-id
  clientSecret: my-client-secret
  cookieSecret: bm90LWdvb2Qtc2VjcmV0Cg==
  enabled: true
  provider: oidc

pinnipedProxy:
  enabled: true

clusters:
  - name: cluster1
    apiServiceURL: https://... # impersonation proxy URL
    certificateAuthorityData: ... #  impersonation proxy CA
    pinnipedConfig:
      enabled: true
  - name:cluster2
    apiServiceURL: https://... # impersonation proxy URL
    certificateAuthorityData: ... #  impersonation proxy CA
    pinnipedConfig:
      enabled: true

When I click on

I see some redirection wheel, and then I am back at the aforementioned Page, so I cannot login.

These are the logs for the auth-proxy

...
192.168.2.3:34874 - badae09224db1af29af8e676f439810f - [email protected] [2022/08/01 14:09:43] kubeapps.my-domain.com GET / "/config.json" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 304 0 0.024
192.168.1.2:42838 - 54574a4a76fa1a24edb3b46df6848933 - - [2022/08/01 14:09:45] kubeapps.my-domain.com GET - "/oauth2/start" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 302 318 0.000
192.168.2.3:34874 - 9f5815b3602734af2deff28c83cc197e - [email protected] [2022/08/01 14:09:46] [AuthSuccess] Authenticated via OAuth2: Session{email:[email protected] user: PreferredUsername: token:true id_token:true created:2022-08-01 14:09:46.456170085 +0000 UTC m=+1215.361068689 expires:2022-08-02 14:09:46.343183348 +0000 UTC m=+87615.248081961}
192.168.2.3:34874 - 9f5815b3602734af2deff28c83cc197e - - [2022/08/01 14:09:46] kubeapps.my-domain.com GET - "/oauth2/callback?code=IJmCZUi2VoUUF46AmLFHAysg2hbRNBXg4C8naSwpA5cyA&state=TavkgOfbVozcBzw8EWsb7W4ryH3Ey7YakGpgBSzAKBg%3A%2F" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 302 24 0.367
192.168.1.2:42838 - 42b472b8c1996b0c02bb3d1b4849dbf9 - [email protected] [2022/08/01 14:10:12] [AuthSuccess] Authenticated via OAuth2: Session{email:[email protected] user: PreferredUsername: token:true id_token:true created:2022-08-01 14:10:12.106140007 +0000 UTC m=+1241.011038610 expires:2022-08-02 14:10:11.999050759 +0000 UTC m=+87640.903949372}
192.168.1.2:42838 - 42b472b8c1996b0c02bb3d1b4849dbf9 - - [2022/08/01 14:10:11] kubeapps.my-domain.com GET - "/oauth2/callback?code=-8MBwsRkCDy77DBZ9sHwGrRj_8YXubPDC3kPo-WSHdpj-&state=JOo6K_lG6SFhP_phE1TPdVKwRA9YIKVUe3vPTP3tCXI%3A%2F" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 302 24 0.229
192.168.2.3:34874 - 27fbf72be060fc25ab0ebce47a1ddc26 - [email protected] [2022/08/01 14:10:12] kubeapps.my-domain.com POST / "/apis/kubeappsapis.plugins.resources.v1alpha1.ResourcesService/CheckNamespaceExists" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 200 0 0.118
192.168.2.3:34874 - 47d5b63a9442e11561a082cd750bd0f1 - [email protected] [2022/08/01 14:11:00] kubeapps.my-domain.com POST / "/apis/kubeappsapis.plugins.resources.v1alpha1.ResourcesService/CheckNamespaceExists" HTTP/1.1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" 200 0 0.982

and these are the logs for the pinniped-proxy:

 INFO  pinniped_proxy > Listening on http://0.0.0.0:3333
 ERROR pinniped_proxy::service > Unauthorized by pinniped: authentication failed

Caused by:
    authentication failed
 INFO  pinniped_proxy::service > GET https://34.141.242.24/api/v1/namespaces/default 401 Unauthorized

It occurs to me as if it is struggling the get the details right with pinniped, but perhaps I am wrong here.

Do you know how to fix that?

[cfryanr]

Unfortunately, I'm not too familiar with debugging Kubeapps. From the auth-proxy logs, it seems like it is getting your identity, because your email address is showing in the log. Maybe @absoludity or @antgamdia could give us some hints?

I'm not sure if the Kubeapps auth-proxy was intended to be used with the Pinniped Supervisor. From your configuration above, I'm guessing that it will go directly to your https://my-domain.eu.auth0.com issuer to log you in and get tokens, and then send that ID token (the one from Auth0) to the Pinniped Concierge on the workload cluster. This seems to imply that the Pinniped Supervisor is cut out of that flow?

[antgamdia]

Thanks, Ryan, for your insights here! We have moved the conversation to vmware-tanzu/kubeapps#5150, so feel free to close this one.

As you pointed out, Kubeapps is not intended to work with the Supervisor but with Pinniped Concierge. So this auth-proxy component is the one talking to the ODIC provider (Auth0 in this case) and performing the underlying required flows with Pinniped Concierge to exchange the k8s token. Obviously, we need to tell Pinniped (via the JWT Authenticator) to trust the OIDC issuer.

For the record, in this step-by-step docs we describe how to do it: https://kubeapps.dev/docs/latest/tutorials/kubeapps-on-tkg/step-1/

[cfryanr]

For anyone interested, more details about potential ways to use both the Pinniped Supervisor and Kubeapps for user authentication at the same time: vmware-tanzu/kubeapps#5150 (reply in thread)