kube-cert-manager not listening on port 8080 for HTTP challenges

[ferrerluis]

I am using v0.3.1 of kube-cert-manager and running GKE on a 1.6.4 k8s cluster.

When I set up my ingress to proxy requests from /.well-known/acme-challenge/* to my kube-cert-manager-service and create a new certificate, LetsEncrypt fails to communicate with the server and find the challenge.

On the Providers Section of the docs, it says that you have to route all connections from /.well-known/acme-challenge/ to the port 8080 for kube-cert-manager.

These are my k8s files:

Deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: kube-cert-manager
  name: kube-cert-manager
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kube-cert-manager
      name: kube-cert-manager
    spec:
      containers:
        - name: kube-cert-manager
          image: palmstonegames/kube-cert-manager:0.3.1
          args:
            - "-data-dir=/var/lib/cert-manager"
            - "-acme-url=https://acme-staging.api.letsencrypt.org/directory"
          volumeMounts:
            - name: data
              mountPath: /var/lib/cert-manager
        - name: kubectl-proxy
          image: palmstonegames/kubectl-proxy:1.4.0
      volumes:
        - name: "data"
          gcePersistentDisk:
            pdName: kube-cert-manager
            fsType: ext4

Service

apiVersion: v1
kind: Service
metadata:
  name: kube-cert-manager-service
spec:
  selector:
    app: kube-cert-manager
  type: NodePort
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080

Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cachet-stage-ingress
  annotations:
    stable.k8s.psg.io/kcm.enabled: "true"
    stable.k8s.psg.io/kcm.provider: "http"
    stable.k8s.psg.io/kcm.email: "[email protected]"
    kubernetes.io/ingress.global-static-ip-name: "cachet-stage-ip"
    kubernetes.io/ingress.class: "gce"
spec:
  # tls:
  # - hosts:
  #   - status-stage.example.com
  #   secretName: status-stage.example.com
  rules:
  - host: "status-stage.example.com"
    http:
      paths:
      - path: /.well-known/acme-challenge/*
        backend:
          serviceName: kube-cert-manager-service
          servicePort: http
  backend:
      serviceName: cachet-stage-service
      servicePort: http

[chrishiestand]

same here, you can see from a netstat within the container that the port 8080 is not open. port 8001 is but does not seem to be usable.

/ # netstat -anlp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8001          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:41412         127.0.0.1:8001          TIME_WAIT   -
tcp        0      0 10.4.1.29:53810         10.7.240.1:443          ESTABLISHED -
tcp        0      0 10.4.1.29:53800         10.7.240.1:443          ESTABLISHED -
tcp        0      0 127.0.0.1:40766         127.0.0.1:8001          ESTABLISHED 1/kube-cert-manager
tcp        0      0 127.0.0.1:41332         127.0.0.1:8001          TIME_WAIT   -
tcp        0      0 127.0.0.1:41492         127.0.0.1:8001          ESTABLISHED 1/kube-cert-manager
tcp        0      0 10.4.1.29:53806         10.7.240.1:443          ESTABLISHED -
tcp        0      0 127.0.0.1:8001          127.0.0.1:40768         ESTABLISHED -
tcp        0      0 127.0.0.1:8001          127.0.0.1:41492         ESTABLISHED -
tcp        0      0 127.0.0.1:40768         127.0.0.1:8001          ESTABLISHED 1/kube-cert-manager
tcp        0      0 127.0.0.1:8001          127.0.0.1:40766         ESTABLISHED -

[tbleich]

I found that configuring an ingress path to the kube-cert-manager-service does not work (I am on IBM Bluemix container service). I am quite new to Kubernetes so I might be mistaken in that point.

Since I am running a Rails and Puma application with a Nginx reverse proxy (=rails+nginx containers in one pod), I configured my nginx containers to forward the path .well-known/acme-challenge/ to the kcm-service. The corresponding part of the nginx.conf looks like this:

location ~ ^/.well-known/acme-challenge/ {
          proxy_pass http://kube-cert-manager-service;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

I suppose only proxy_pass and the Host header are really necessary.
Using KCM version 0.5 from image iosphere/kube-cert-manager:c8443d4.

@ferrerluis not entirely sure, but in order for a Kubernetes service to work properly, don't you have to open port 8080 in your deployment?