How to termination of mTLS, and use tcp to connect to EMQX

[Rory-Z]

Hi, guys
I use EMQX Operator to deploy EMQX like this:

$ cat << "EOF" | kubectl apply -f -        
apiVersion: apps.emqx.io/v1beta3
kind: EmqxEnterprise
metadata:
  name: emqx-ee
spec:    
  emqxTemplate:
    image: emqx/emqx-ee:4.4.5
EOF

Now, I want termination mTLS, and use tcp to connect to EMQX, have any example ?

[Gala-R]

I have an example that will solve your problem

[Gala-R]

Deploy HAProxy on K8S, terminate mTLS connection

Goals

• Terminate mTLS on the HAProxy side

• mTLS certificate mounted to HAProxy container using tls secret

• haproxy.cfg mounted to the HAProxy container using configmap

• HAProxy's backend service uses DNS name to route to EMQX pods

• EMQX cluster is deployed using Operator

• Enable proxy protocol v2, pass the CN field of the certificate to EMQX

Environment

• HAProxy 1.8

• EMQX Operator 1.2.2

• Kubernetes 1.24

Configuration

• The deployment of EMQX Operator uses the emqx-operator-controller.yaml file

• mTLS configuration reference tls.yaml file

• haproxy.cfg of EMQX Enterprise reference haproxy-cfg.yaml

• HAProxy is deployed in k8s using deployment, the deployment file refers to haproxy.yaml

• The deployment of EMQX Enterprise uses the emqx-ee.yaml file

Architecture

Process

• Create tls secret and haproxy.cfg configMap

  cat << EOF | kubectl apply -f -
  apiVersion: v1
  data:
    ca.crt: 
    tls.crt: 
    tls.key: 
  kind: Secret 
  metadata:
    name: haproxytls
    namespace: default
  type: kubernetes.io/tls
  EOF
  

  cat << EOF | kubectl apply -f - 
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: haproxy.cfg
  data:
      haproxy.cfg: |-
              global  
                    log 127.0.0.1 local3 info 
                    #chroot /opt/apps/haproxy 
                    daemon  
                    maxconn 1024000 
              defaults  
                    log global 
                    mode tcp 
                    option tcplog 
                    #option dontlognull  
                    timeout connect 10000 
                    # timeout > mqtt's keepalive * 1.2  
                    timeout client 240s  
                    timeout server 240s 
                    # 'full transparent proxy' for all 
                    #source 0.0.0.0 usesrc clientip  
                    maxconn 20000 
              frontend emqx_ssl 
                    bind *:8883 ssl  ca-file /usr/local/etc/haproxy/tls/ca.crt crt /usr/local/etc/haproxy/tls/tls.crt  verify required
                    mode tcp  
                    default_backend emqx_ssl_back 
  
              backend emqx_ssl_back 
                    mode tcp  
                    balance roundrobin 
                    server emqx-ee-0  emqx-ee-0.emqx-ee-headless.default.svc.cluster.local:1883 send-proxy-v2-ssl-cn check 
                    server emqx-ee-1  emqx-ee-1.emqx-ee-headless.default.svc.cluster.local:1883 send-proxy-v2-ssl-cn check 
                    server emqx-ee-2  emqx-ee-2.emqx-ee-headless.default.svc.cluster.local:1883 send-proxy-v2-ssl-cn check 
  EOF
  

• Deploy emqx-operator-controller and EMQX cluster

  kubectl apply -f "https://github.com/emqx/emqx-operator/releases/download/1.2.2/emqx-operator-controller.yaml"
  

  cat << EOF | kubectl apply -f -
  apiVersion: apps.emqx.io/v1beta3
  kind: EmqxEnterprise
  metadata:
    name: emqx-ee
    labels:
      "foo": "bar"
  spec:
    emqxTemplate:
      image: emqx/emqx-ee:4.4.5
      config:
          # If the EMQX cluster is deployed behind HAProxy or Nginx, and you need to get the real source IP address and port of the client, you need to open this configuration.
          listener.tcp.external.proxy_protocol: "on"
          # Use the client certificate to override the value of the Username field
          listener.tcp.external.peer_cert_as_username: "cn"
  
  EOF
  

• Deploy HAProxy

  cat << EOF | kubectl -f - 
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: haproxy
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: haproxy
    template:
      metadata:
        labels:
          app: haproxy
      spec:
        containers:
        - name: haproxy
          image: haproxy:1.8
          ports:
          - name: tcp
            containerPort: 80
          volumeMounts:
          - mountPath: /usr/local/etc/haproxy/
            name: haproxy-cfg
            readOnly: true 
          - mountPath: /usr/local/etc/haproxy/tls/
            name: haproxy-tls 
            readOnly: true 
        volumes:
        - name: haproxy-cfg
          configMap:
            name: haproxy.cfg
        - name: haproxy-tls 
          secret:
            secretName: haproxytls 
  EOF
  

• Observe whether the EMQX cluster and HAProxy are running normally

  kubectl get pods -n default 
  

• Based on MQTTX, verify whether the message can be sent to EMQX normally, and correctly pass the CN field of the certificate to EMQX

  kubectl exec -it emqx-ee-0  -- bash -c "curl -i --basic -u admin:public -X GET http://127.0.01:8081/api/v4/clients"  
  

  Use the above command to check whether the value of the username field in the returned data is consistent with the CN of the client certificate.

Remark

• How to modify the HProxy configuration file and make it take effect after the EMQX node is scaled

  After scaling the EMQX node, you need to modify the backend field of the haproxy-cfg.yaml file.

  For example, if the emqx-3 node is added, add the following information in the haproxy.cfg file:

  server emqx-ee-3  emqx-ee-3.emqx-ee-headless.default.svc.cluster.local:1883 send-proxy-v2-ssl-cn check 
  

  Then use the kubectl apply -f haproxy-cfg.yaml command to update the ConfigMap, enter the HAProxy container and check whether the haproxy.cfg configuration has been updated.

• How to renew HAProxy certificates

  After updating the certificate, update the data of the three fields of ca.crt, tls.crt and tls.key in the tls.yaml file with the latest certificate data.

  Then use the kubectl apply -f tls.yaml command to update the Secret, and enter the HAProxy container to check whether the ca.crt, tls.crt, and tls.key data have been updated.

• What is Proxy protocol

  The proxy protocol is the IP of HAProxy, an Internet protocol designed by Wly Tarreau in 2010. By adding the header information of a protocol development stack to the source port, it is convenient to transmit, source IP, destination author, port information, destination port, etc., It is very useful when you need to obtain the user's real IP in the case of a complex network) .

• listener.tcp.external.peer_cert_as_username

  Type	Optional Value	Default
  enum	cn	cn

  Use a certificate to override the value of the username field.

  Note: Under the TCP listener, this configuration can only be used when the load balancing server terminates SSL deployment; and the load balancing server needs to configure Proxy Protocol to send the content of the certificate domain to EMQX.

• listener.tcp.external.proxy_protocol

  Type	Optional Value	Default
  enum	on, off	-

  Whether the listener enables Proxy Protocol support.

  If the EMQX cluster is deployed behind HAProxy or Nginx, and you need to get the real source IP address and port of the client, you need to open this configuration.

[xtianus79]

@Gala-R so I have the TCP working well now with the ingress controller.

I'll show you. I feel like I'm almost there and if I could get over this last part I would be there.

with this I am able to set the TCP and a config is created shown below. I use letsencrypt RA cert and it does work on a direct connection to the port 8883. The issue is. I would imagine for the last config below I would be able to switch this port:

from this
'8883': ingress-emqx/emqx-ee:8883

to this
'8883': ingress-emqx/emqx-ee:1883

But when I do that I get this error while using mqttx with the setting I have working otherwise with a direct connect of CA signed server / SSL Secure toggled off: If I toggle SSL secure on it gives 2 of the same error at the same time. Any help getting the rest of the way would be amazing.
Error: Client network socket disconnected before secure TLS connection was established

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --namespace $NAMESPACE --set controller.service.annotations.service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path=/healthz,controller.service.annotations.service\.beta\.kubernetes\.io/azure-dns-label-name=$DNS_LABEL --set controller.service.loadBalancerIP=$STATIC_IP --set tcp.18083=$NAMESPACE/emqx-ee:18083,tcp.8883=$NAMESPACE/emqx-ee:8883

My ingress-ngnix-controller is this:

spec:
  ports:
    - name: http
      protocol: TCP
      appProtocol: http
      port: 80
      targetPort: http
      nodePort: 31404
    - name: https
      protocol: TCP
      appProtocol: https
      port: 443
      targetPort: https
      nodePort: 30323
    - name: 18083-tcp
      protocol: TCP
      port: 18083
      targetPort: 18083-tcp
      nodePort: 32103
    - name: 8883-tcp
      protocol: TCP
      port: 8883
      targetPort: 8883-tcp
      nodePort: 30736

And lastly my config map is this.

kind: ConfigMap
apiVersion: v1
metadata:
  name: ingress-nginx-tcp
  namespace: ingress-emqx
  uid: 53440df5-f948-4e3b-8291-f534bf6aecb4
  resourceVersion: '2678198'
  creationTimestamp: '2022-08-26T20:35:54Z'
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.3.0
    helm.sh/chart: ingress-nginx-4.2.3
  annotations:
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: ingress-emqx
  managedFields:
    - manager: helm
      operation: Update
      apiVersion: v1
      time: '2022-08-31T04:53:52Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:data:
          .: {}
          f:18083: {}
        f:metadata:
          f:annotations:
            .: {}
            f:meta.helm.sh/release-name: {}
            f:meta.helm.sh/release-namespace: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/component: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:app.kubernetes.io/part-of: {}
            f:app.kubernetes.io/version: {}
            f:helm.sh/chart: {}
    - manager: Mozilla
      operation: Update
      apiVersion: v1
      time: '2022-09-01T07:14:00Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:data:
          f:8883: {}
data:
  '8883': ingress-emqx/emqx-ee:8883
  '18083': ingress-emqx/emqx-ee:18083

[xtianus79]

@Gala-R I can't get this working because I am getting this error:

'bind *:8883' : unable to load SSL private key from PEM file '/usr/local/etc/haproxy/tls/tls.crt'

What is the structure of the 3 values you have listed?

cat << EOF | kubectl apply -f -
apiVersion: v1
data:
  ca.crt: <key>
  tls.crt:  <key>
  tls.key:  <key>
kind: Secret 
metadata:
  name: haproxytls
  namespace: default
type: kubernetes.io/tls
EOF

I don't see why it would complain about that if there is a key included as one of the options. I also can't find anywhere on the internet where the setup is similar to this.

[xtianus79]

Hi @Gala-R I have an idea. But can you help me please. It would be very beneficial to Azure users and it would go a long way to making the docs a little better. Better docs = higher adoption rate and this is a fact. I would like to transform this to letsencrypt and the Certificate type of key generation.

[xtianus79]

@Gala-R so I don't think this solution you've posted works for Azure AKS at all. I have done something with Haproxy ingress. Which seems to be a better ingress for mqtt than nginx.

I would post my solution and findings in another post.

TLDR. I think it is ssl offloading but not sure. In the dashboard I am seeing a 1 connection on both the 8883 and 1883 tcp. I think that means it is offloading but not sure why both ports are being detected in the dashboard. Could make perfect sense.

[xtianus79]

I figured it out. You got me in the right direction. Thanks @Gala-R