Knowledge Base
linbit.com Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage
  1. Knowledge Base Topics
  2. /
  3. LINSTOR
  4. /
  5. Kubernetes
  6. /
  7. Controlling Replicas With Kubernetes Node Labels and LINSTOR Auxiliary Properties

Controlling Replicas With Kubernetes Node Labels and LINSTOR Auxiliary Properties

By using Kubernetes node labels and LINSTOR® auxiliary properties, you can better control the placement of your replicas within your cluster. This is useful when you need to avoid placing two replicas within a single failure domain (such as a rack or DC).

Assume that you have a six node Kubernetes cluster with LINSTOR configured using the LINSTOR Operator for persistent storage, and you have a LINSTOR storage pool named lvm-thin configured across all nodes.

# kubectl get nodes
NAME     STATUS   ROLES           AGE     VERSION
kube-0   Ready    control-plane   6h57m   v1.26.3
kube-1   Ready    <none>          6h57m   v1.26.3
kube-2   Ready    <none>          6h57m   v1.26.3
kube-3   Ready    <none>          6h57m   v1.26.3
kube-4   Ready    <none>          6h57m   v1.26.3
kube-5   Ready    <none>          6h57m   v1.26.3

LINSTOR ==> node list
╭───────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Node                                      ┊ NodeType   ┊ Addresses                   ┊ State  ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════╡
┊ kube-0                                    ┊ SATELLITE  ┊ 192.168.222.40:3366 (PLAIN) ┊ Online ┊
┊ kube-1                                    ┊ SATELLITE  ┊ 192.168.222.41:3366 (PLAIN) ┊ Online ┊
┊ kube-2                                    ┊ SATELLITE  ┊ 192.168.222.42:3366 (PLAIN) ┊ Online ┊
┊ kube-3                                    ┊ SATELLITE  ┊ 192.168.222.43:3366 (PLAIN) ┊ Online ┊
┊ kube-4                                    ┊ SATELLITE  ┊ 192.168.222.44:3366 (PLAIN) ┊ Online ┊
┊ kube-5                                    ┊ SATELLITE  ┊ 192.168.222.45:3366 (PLAIN) ┊ Online ┊
┊ linstor-op-cs-controller-7c7d59d98d-d82lr ┊ CONTROLLER ┊ 172.16.186.2:3366 (PLAIN)   ┊ Online ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────╯

LINSTOR ==> storage-pool list
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node   ┊ Driver   ┊ PoolName          ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
8<------------------------------------------------------------snip---------------------------------------------------------------8<
┊ lvm-thin             ┊ kube-0 ┊ LVM_THIN ┊ drbdpool/thinpool ┊     8.00 GiB ┊      8.00 GiB ┊ True         ┊ Ok    ┊            ┊
┊ lvm-thin             ┊ kube-1 ┊ LVM_THIN ┊ drbdpool/thinpool ┊     8.00 GiB ┊      8.00 GiB ┊ True         ┊ Ok    ┊            ┊
┊ lvm-thin             ┊ kube-2 ┊ LVM_THIN ┊ drbdpool/thinpool ┊     8.00 GiB ┊      8.00 GiB ┊ True         ┊ Ok    ┊            ┊
┊ lvm-thin             ┊ kube-3 ┊ LVM_THIN ┊ drbdpool/thinpool ┊     8.00 GiB ┊      8.00 GiB ┊ True         ┊ Ok    ┊            ┊
┊ lvm-thin             ┊ kube-4 ┊ LVM_THIN ┊ drbdpool/thinpool ┊     8.00 GiB ┊      8.00 GiB ┊ True         ┊ Ok    ┊            ┊
┊ lvm-thin             ┊ kube-5 ┊ LVM_THIN ┊ drbdpool/thinpool ┊     8.00 GiB ┊      8.00 GiB ┊ True         ┊ Ok    ┊            ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Also assume you have your six nodes evenly distributed across three separate racks within your data center, or across three separate availability zones (AZ) within a cloud region. In our examples, we’ll assume kube-0 and kube-1 are in one rack or AZ, kube-2 and kube-3 are in another, and kube-4 and kube-5 are in yet another.

LINSTOR, by default, is not aware of this distribution and therefore might place both replicas of a two replica LINSTOR volume within the same rack or AZ. This would leave your data inaccessible during a rack or AZ outage. Alternatively, you might want to keep replicas within a single rack or AZ to isolate LINSTOR’s replication, or to keep replication latency to an absolute minimum.

In either situation, you will first need to add Kubernetes labels to each node. The LINSTOR Operator will automatically import Kubernetes node labels into LINSTOR and apply them as auxiliary properties on the LINSTOR node objects. Using the assumptions above, you will add the following node labels to your Kubernetes nodes, using the key zone with values a, b, and c to differentiate your racks or AZs.

# kubectl label nodes kube-{0,1} zone=a
node/kube-0 labeled
node/kube-1 labeled

# kubectl label nodes kube-{2,3} zone=b
node/kube-2 labeled
node/kube-3 labeled

# kubectl label nodes kube-{4,5} zone=c
node/kube-4 labeled
node/kube-5 labeled

You’ll see the Kubernetes node labels on each of the respective LINSTOR node objects.

LINSTOR ==> node list-properties kube-0
╭────────────────────────────────────────────────────────────────────────────────╮
┊ Key                                                         ┊ Value            ┊
╞════════════════════════════════════════════════════════════════════════════════╡
┊ Aux/beta.kubernetes.io/arch                                 ┊ amd64            ┊
┊ Aux/beta.kubernetes.io/os                                   ┊ linux            ┊
┊ Aux/kubernetes.io/arch                                      ┊ amd64            ┊
┊ Aux/kubernetes.io/hostname                                  ┊ kube-0           ┊
┊ Aux/kubernetes.io/os                                        ┊ linux            ┊
┊ Aux/linbit.com/hostname                                     ┊ kube-0           ┊
┊ Aux/linbit.com/sp-DfltDisklessStorPool                      ┊ true             ┊
┊ Aux/linbit.com/sp-lvm-thick                                 ┊ true             ┊
┊ Aux/linbit.com/sp-lvm-thin                                  ┊ true             ┊
┊ Aux/node-role.kubernetes.io/control-plane                   ┊                  ┊
┊ Aux/node.kubernetes.io/exclude-from-external-load-balancers ┊                  ┊
┊ Aux/registered-by                                           ┊ linstor-operator ┊
┊ Aux/zone                                                    ┊ a                ┊
┊ CurStltConnName                                             ┊ default          ┊
┊ NodeUname                                                   ┊ kube-0           ┊
╰────────────────────────────────────────────────────────────────────────────────╯

Placing Replicas in Different Zones

LINSTOR’s storageClasses can then be configured to avoid placing replicas within a single failure domain using the LINSTOR storageClass parameter replicasOnDifferent, naming the zone key.

cat << EOF > linstor-sc-on-diff.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-csi-lvm-thin-r2-on-diff
provisioner: linstor.csi.linbit.com
parameters:
  autoPlace: 2
  storagePool: lvm-thin
  replicasOnDifferent: zone
reclaimPolicy: Delete
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-csi-lvm-thin-r3-on-diff
provisioner: linstor.csi.linbit.com
parameters:
  autoPlace: 3
  storagePool: lvm-thin
  replicasOnDifferent: zone
reclaimPolicy: Delete
allowVolumeExpansion: true
EOF

kubectl apply -f linstor-sc-on-diff.yaml

Creating persistent volume claims (PVC) using the storageClass created above will result in replicas being distributed where the key zone has different values.

cat << EOF > pvcs-on-diff.yaml
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: demo-vol-claim-diff-zone-0
spec:
  storageClassName: linstor-csi-lvm-thin-r3-on-diff
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1G
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: demo-vol-claim-diff-zone-1
spec:
  storageClassName: linstor-csi-lvm-thin-r3-on-diff
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1G
EOF

kubectl apply -f pvcs-on-diff.yaml

Within LINSTOR, you will see that each replica of the LINSTOR resources is in a different zone.

LINSTOR ==> resource list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node   ┊ Port ┊ Usage  ┊ Conns ┊    State ┊ CreatedOn           ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-c38af6c1-f02a-46db-b8ac-74b4eef20ca6 ┊ kube-0 ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:39:18 ┊
┊ pvc-c38af6c1-f02a-46db-b8ac-74b4eef20ca6 ┊ kube-2 ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:39:17 ┊
┊ pvc-c38af6c1-f02a-46db-b8ac-74b4eef20ca6 ┊ kube-4 ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:39:18 ┊
┊ pvc-e8a5d0c8-9e61-46c3-afb5-f0ca975c4249 ┊ kube-1 ┊ 7001 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:39:17 ┊
┊ pvc-e8a5d0c8-9e61-46c3-afb5-f0ca975c4249 ┊ kube-3 ┊ 7001 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:39:19 ┊
┊ pvc-e8a5d0c8-9e61-46c3-afb5-f0ca975c4249 ┊ kube-4 ┊ 7001 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:39:19 ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Placing Replicas in the Same Zone

LINSTOR’s storageClasses can also be configured to place replicas within the same zone by using the LINSTOR storageClass parameter replicasOnSame, naming the respective key and value pair.

cat << EOF > linstor-sc-on-same.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-csi-lvm-thin-r2-on-same-a
provisioner: linstor.csi.linbit.com
parameters:
  autoPlace: 2
  storagePool: lvm-thin
  replicasOnSame: zone=a
reclaimPolicy: Delete
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-csi-lvm-thin-r2-on-same-b
provisioner: linstor.csi.linbit.com
parameters:
  autoPlace: 2
  storagePool: lvm-thin
  replicasOnSame: zone=b
reclaimPolicy: Delete
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-csi-lvm-thin-r2-on-same-c
provisioner: linstor.csi.linbit.com
parameters:
  autoPlace: 2
  storagePool: lvm-thin
  replicasOnSame: zone=c
reclaimPolicy: Delete
allowVolumeExpansion: true
EOF

kubectl apply -f linstor-sc-on-same.yaml

Creating PVCs using the storageClasses created above will result in replicas being distributed where the key zone has the specified value.

cat << EOF > pvcs-on-same.yaml
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: demo-vol-claim-zone-a
spec:
  storageClassName: linstor-csi-lvm-thin-r2-on-same-a
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1G
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: demo-vol-claim-zone-b
spec:
  storageClassName: linstor-csi-lvm-thin-r2-on-same-b
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1G
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: demo-vol-claim-zone-c
spec:
  storageClassName: linstor-csi-lvm-thin-r2-on-same-c
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1G
EOF

kubectl apply -f pvcs-on-same.yaml

Within LINSTOR, you will see that each replica of the LINSTOR resources are in the same zone.

LINSTOR ==> resource list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node   ┊ Port ┊ Usage  ┊ Conns ┊    State ┊ CreatedOn           ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-0ef85bf7-2a9a-4e6f-9d7b-a473518c6cee ┊ kube-2 ┊ 7001 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:17:55 ┊
┊ pvc-0ef85bf7-2a9a-4e6f-9d7b-a473518c6cee ┊ kube-3 ┊ 7001 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:17:52 ┊
┊ pvc-0fc56b3d-b249-4e6f-a225-41224cb367f9 ┊ kube-0 ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:17:52 ┊
┊ pvc-0fc56b3d-b249-4e6f-a225-41224cb367f9 ┊ kube-1 ┊ 7000 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:17:53 ┊
┊ pvc-35144a76-d15f-4709-9911-b6c951e87cc1 ┊ kube-4 ┊ 7002 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:17:54 ┊
┊ pvc-35144a76-d15f-4709-9911-b6c951e87cc1 ┊ kube-5 ┊ 7002 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2023-03-24 22:17:56 ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Written by: MDK - 3/24/23