A question about emqx cluster core node persistent storage

[neverlaughtingboy]

I want to deploy emqx cluster on aliyun, adopting the method of core node and copy node. I am more confused because many tutorials give different examples for persistence. For example, in the helm deployment method, you can directly use your own pre-set pvc, in the public cloud platform deployment guide, let us use the specified alibabacloud-cnfs-nas to automatically create pv and pvc, and configure the EMQX persistence task is another way. My problem is that the data persisted by the core node is the node It is enough to mount a pvc shared among them, or they should manage their own. Do you have any suggestions for the persistence of core nodes?

[Rory-Z]

Maybe you can check https://docs.emqx.com/en/emqx-operator/latest/tasks/configure-emqx-persistence.html

[badsector3]

Hi. Instead of creating the new Q&A I thought I will just write here.

I followed the configuration above. The problem which I see is as following:

• I use EFS

      volumeClaimTemplates:
        storageClassName: standard
        resources:
          requests:
            storage: 20Mi
        accessModes:
          - ReadWriteOnce

All good so far but if I restart the EMQX (I use SPOT instances, so the restart will happen) - new PVC is created and the PVC will create new PV - instead of the reusing the already created PV.

EMQX operator chart version: 2.2.5
CRD:

apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
  name: {{ include "emqx-cluster.fullname" . }}
  labels:
     {{- include "emqx-cluster.labels" . | nindent 4 }}
spec:
  config:
    data: {{ .Files.Get "files/emqx-config.yaml" | quote }}
  image: {{ .Values.emqx.image.repository }}:{{ .Values.emqx.image.tag }}
  coreTemplate:
    metadata:
      labels: 
        app.kubernetes.io/name: {{ template "emqx-cluster.fullname" . }}
        {{- range .Values.securityGroupPolicies }}
        {{ . | quote }}: "true"
        {{- end }}
    spec: 
      replicas: {{ .Values.coreTemplate.replicas }}
      volumeClaimTemplates:
        storageClassName: emqx-storage-class
        volumeMode: Filesystem
        resources:
          requests:
            storage: 20Gi
        accessModes:
          - ReadWriteMany

My storageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  labels:
    name: emqx-sc
parameters:
  basePath: /emqx-core
  directoryPerms: "700"
  fileSystemId: fs-xxxxxxxxxxx
  gidRangeEnd: "2000"
  gidRangeStart: "1000"
  provisioningMode: efs-ap
provisioner: efs.csi.aws.com
reclaimPolicy: Retain
volumeBindingMode: Immediate

I think I should set the "ensureUniqueDirectory=false" as described here: https://github.com/kubernetes-sigs/aws-efs-csi-driver/tree/master

Any ideas?

[Rory-Z]

I'm sorry, I am not an AWS expert, perhaps you can expect answers from others.

[badsector3]

Thanks @Rory-Z

Adding that ensureUniqueDirectory=false flag didnt fix the issue. I had to take the storageClass away from Helm chart and move it to IaC, passing the storageClassName only.

So far looks to be working fine. Full restart of the Helm will not create new PVC because storageClass is managed in IaC.