Unable to provision volumes on all nodes with StorageClass with Custom Node Labels

[khalMeg]

I am using openEBS localpv LVM with a StorageClass with Custom Node Labels (for all my worker nodes). However, volumes are always provisioned on node worker-01, and if I cordon worker-01 volumes will never be scheduled on different nodes.

workaround : if I create a new storage class for a specific node (other than worker-01) the volume is provisioned without any problem.

here is my storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: main-sc
annotations:
allowVolumeExpansion: true
parameters:
  thinProvision: "yes"
  storage: "lvm"
  volgroup: "data"
  fsType: "xfs"
provisioner: local.csi.openebs.io
allowedTopologies:
- matchLabelExpressions:
  - key: kubernetes.io/hostname
    values:
      - worker-01
      - worker-02
      - worker-03
      - worker-04
     

• kubectl logs -f openebs-lvm-localpv-controller-7b6d6b4665-fk78q -n openebs -c openebs-lvm-plugin
  https://gist.github.com/khalMeg/ff579367b9ad5c0d5016841b3607aa8a

• kubectl logs -f openebs-lvm-localpv-node-[xxxx] -n openebs -c openebs-lvm-plugin
  https://gist.github.com/khalMeg/cd740b62d5102d61e99160bbb8212cac

kubectl get pods -n openebs
NAME                                             READY   STATUS    RESTARTS       AGE
openebs-localpv-provisioner-6bd66f8598-nnzzv     1/1     Running   2 (4d3h ago)   17d
openebs-lvm-localpv-controller-6bbd64786-kvbw4   5/5     Running   0              17d
openebs-lvm-localpv-node-2plnt                   2/2     Running   8 (17d ago)    39d
openebs-lvm-localpv-node-bw68q                   2/2     Running   8 (17d ago)    39d
openebs-lvm-localpv-node-nb7wl                   2/2     Running   0              38h
openebs-lvm-localpv-node-vjpgb                   2/2     Running   0              13d
openebs-zfs-localpv-controller-b8bc9ff46-x9k9w   5/5     Running   0              17d
openebs-zfs-localpv-node-bmgl6                   2/2     Running   13 (17d ago)   39d
openebs-zfs-localpv-node-lsh8p                   2/2     Running   9 (17d ago)    39d
openebs-zfs-localpv-node-n6p8p                   2/2     Running   0              38h
openebs-zfs-localpv-node-r9lmd                   2/2     Running   0              13d

Environment:

• LVM Driver version

LVM version:     2.03.16(2) (2022-05-18)
Library version: 1.02.185 (2022-05-18)
Driver version:  4.47.0

• Kubernetes version :

Client Version: v1.30.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.4

• Kubernetes installer & version:
  k8s cluster deployed using Kubespray
• OS:
  Debian GNU/Linux 12 (bookworm)

[abhilashshetty04]

Hi @khalMeg ,

Can you share the new storage class with which provisioning works for other node.

Please also see the docs for custom label in storage class

https://openebs.io/docs/user-guides/local-storage-user-guide/local-pv-lvm/lvm-configuration#storageclass-with-custom-node-labels

[khalMeg]

Hi @abhilashshetty04
It is the same storageClass only with different node selected. This is the only way I can go about provisioning on nodes other than worker-01.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: w2-sc
allowVolumeExpansion: true
parameters:
  thinProvision: "yes"
  storage: "lvm"
  volgroup: "data"
  fsType: "xfs"
provisioner: local.csi.openebs.io
allowedTopologies:
- matchLabelExpressions:
  - key: kubernetes.io/hostname
    values:
       - worker-02

The same issue occurs when I use a storage class without specifying any node; the volumes are always provisioned on worker-01!

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: main-sc
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
allowVolumeExpansion: true
parameters:
  thinProvision: "yes"
  storage: "lvm"
  volgroup: "data"
  fsType: "xfs"
provisioner: local.csi.openebs.io

Please note that I am using Thin Provisioning in my LVM setup.