Pinniped TLS handshake with ingress

[HamzaZo]

Hello,
I'm implementing Pinniped with our on-premise Kubernetes cluster. I have deployed different versions of Pinniped (0.21.0 or even the latest version), but I am encountering strange issues related to TLS handshake errors between Pinniped-supervisor and ingress.

I am using cert-manager to create a certificate for the supervisor and exposing it with ingress on port 443 of the Pinniped-supervisor-api service. I have also added the ingress annotations 'nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"'. However, when I try a simple curl -v https://my-pinniped-supervisor/, I am getting a 502 error. When I check the logs of the ingress, I am seeing a lot of errors related to TLS handshake.

Any ideas, please?

[cfryanr]

Hi @HamzaZo, thanks for submitting a question!

Your Ingress should not use the pinniped-supervisor-api Service as its backend. That Service was added to recent versions of the Pinniped Supervisor, and is used for a different purpose (it helps to serve an aggregated API to the Kubernetes API server. Instead, you should create your own Service which exposes the HTTPS port 8443 of the Supervisor pods, and use that Service as the backend of your Ingress. See https://pinniped.dev/docs/howto/configure-supervisor/ for more information.

Does that help?

[HamzaZo]

Hi @cfryanr I appreciate your reply, I'll try that !

When I use ldapidentityprovider with our LDAP, I'm getting error about TLS protocol.

Does ldapidentityprovider use TLSv1.3 by default?

Thanks

[cfryanr]

Hi @HamzaZo,

The code sets MinVersion: tls.VersionTLS12 in the Golang standard library's tls.Config struct. So I think that means that it will negotiate with the server to together choose either TLS 1.2 or TLS 1.3, based on what the server supports.

However, note that the code also limits which TLS cipher suites are to be allowed, to avoid cipher suites that are known to be insecure. You can see that list in this file: https://github.com/vmware-tanzu/pinniped/blob/v0.23.0/internal/crypto/ptls/default.go. Is your LDAP server perhaps not offering any of the allowed cipher suites?

If you'd like to trace through all the details of the LDAP client's TLS code yourself, you can start where the LDAP client code does its dialing in these functions: https://github.com/vmware-tanzu/pinniped/blob/v0.23.0/internal/upstreamldap/upstreamldap.go#L299-L344

Does that help?

[HamzaZo]

Hi @cfryanr
I created my own Service which exposes the HTTPS port 8443 of the Supervisor pods, and I use that Service as the backend of my Ingress. however I'm getting pinniped supervisor has invalid TLS serving certificate configuration I see in the code that is related to bootsrap func

pinniped/internal/supervisor/server/bootstrap.go

Line 44 in 60d12d8

http.Error(w, "pinniped supervisor has invalid TLS serving certificate configuration", http.StatusInternalServerError)

Any idea why I'm getting this ? btw I'm using cert-manager to create cert for my ingress

[EDIT]
I downgraded the pinniped version to v0.12.1 in order to validate that everything works as expected with Dex as IDP due to LDAP TLS protocol. I set up everything and when I try kubectl get po -A --kubeconfig pinniped-kubeconfig

Pinniped open up the browser, but it only displayed a blank page. I noticed an issue with that the Redirect_URL is localhost https://my-ingress/oauth2/authorize?access_type=offline&client_id=pinniped-cli&code_challenge=xxxxx&code_challenge_method=S256&nonce=xxxxx&pinniped_idp_name=dex&pinniped_idp_type=oidc&redirect_uri=http://127.0.0.1:39591/callback...

[cfryanr]

Hi @HamzaZo, are you using a DNS hostname for the Supervisor, or are you using an IP address to access the Supervisor?

If you are trying to use a DNS name, when your Ingress makes the HTTPS request to its backend, does it preserve the hostname of the original request or set the Host header to pass-through the hostname of the original request? The Supervisor uses SNI (hostname on the request or Host header on the request) to find the TLS certificate configured on your FederationDomain. To help debug, you can turn on trace logging for the Supervisor and then watch for this log message during a request: https://github.com/vmware-tanzu/pinniped/blob/v0.23.0/internal/supervisor/server/server.go#L530-L535.

If you are using an IP address to make requests to the Supervisor, or if your Ingress cannot be configured to properly preserve SNI on requests, then you can put the Supervisor's TLS serving certificate into a Secret at the name/namespace location for the default serving cert. This will be used as a fallback whenever a request comes in to an IP address or without SNI information.

Does that help?

[HamzaZo]

Hi @cfryanr

I appreciate your reply, I'm using a DNS hostname for the Supervisor. I'll enable the trace logging on it to see what's happening and let you know

[HamzaZo]

Hi @cfryanr

I tested it and it works fine, thanks a lot! However, I'm encountering an issue with OIDCIdentityProvider after successfully connecting to LDAP through Dex. I'm getting an error message that says Unprocessable Entity: required claim in upstream ID token missing.

I turned on debug logging on Dex, and I can see all the information being passed username: toto, groups: [group1], email: my-email.

I'm using the following configuration

kind: OIDCIdentityProvider
metadata:
  namespace: pinniped-supervisor
  name: dex
spec:
  issuer: https://my-issuer-dex/dex
  authorizationConfig:
    additionalScopes: [openid, profile, groups, email, username]
    allowPasswordGrant: false 
  claims:
    username: email
  client:
    secretName: dex-client-credentials

---
apiVersion: v1
kind: Secret
metadata:
  namespace: pinniped-supervisor
  name: dex-client-credentials
type: secrets.pinniped.dev/oidc-client
stringData:
  clientID: "xxxxx"
  clientSecret: "xxxxx"

any idea why I'm getting claim errors ?

[cfryanr]

I turned on debug logging on Dex, and I can see all the information being passed username: toto, groups: [group1], email: my-email.
I'm sorry, I didn't understand what you meant by this. Are these the claims in the Dex ID token?

By the way, if your goal is to use LDAP, it should be easier to use Pinniped's direct support for LDAP (LDAPIdentityProvider or ActiveDirectoryIdentityProvider) rather than using Dex to connect to LDAP.

[cfryanr]

By the way, did you figure out what was the problem that you were having with TLS to your LDAP server when using LDAPIdentityProvider? What was the error message that you were seeing?

[HamzaZo]

For the moment, I'm using Dex to bypass the TLS protocol until we upgrade to the newest version of LDAP. I believe that our LDAP only supports an older version of TLS.

The message was: The server selected an unsupported version of TLS Protocol