To learn how to configure NFS with RWX access-mode in a KIND Kubernetes Cluster

There are certain ways to create Kubernetes cluster locally for testing and learning purposes through various tools such as docker desktop, minikube, orkind and each of them comes with different features like minikube and docker desktop allows to create a single-node cluster whereas kind allows creating a multi-node cluster.

As Kind allows to create multi-node cluster locally and comes with a default standard storage class(ReadWriteOnce access mode) which sometimes may not be useful like consider a scenario when one wants to use pods on multiple nodes for read and write operations with the same persistent volume but can not do it because of ReadWriteOnce access mode.

Persistent Volume in Kubernetes

Basically, with the persistent volume(PV), we do provisioning of storage for applications running in Pods, either manually through cluster-admin (static volume provisioning) or dynamically through storage classes (dynamic volume provisioning).

These storage classes allow admins to provide different classes of storage through different types of provisioners such as NFS, EBS, and many more. And to use this resource, it must be requested through Persistent Volume Claim(PVC). A PVC object allows pods to use storage from persistent volumes.

Dynamic volume provisioning allows storage class objects to define which provisioner to be used as these provisioners set the different access modes for persistent volume to be used.

Access Modes in Persistent Volume

Access modes are the way to tell us about different modes of operations each PV holds and they are 

• ReadWriteOnce (RWO): It allows the volume to be mounted on a single node with Read and Write permissions but it allows multiple pods to access the same volume running on the same node.
• ReadWriteMany (RWX): It allows volume can be mounted by multiple nodes with Read and Write permissions and also allow multiple pods to access the same volume.
• ReadOnlyMany (ROX): It allows the volume to be mounted by multiple nodes but with Read permissions only and allows multiple pods to access the same volume to Read the data.
• ReadWriteOncePod (RWOP ): It allows the volume to be mounted by a single pod in the cluster with Read and Write permissions.

Configuring kind cluster with its default storage class

To deep dive into more, first, we will be going to create a kubernetes cluster with kind as it is easy to configure.

•  The following cluster will have two worker nodes and one control-plane(master) node.

kind create cluster --config=kind-config.yaml

 It will take a few minutes to get the kind cluster configured and as it gets configured, the Ready status of nodes can be verified by

kubectl get nodes

• Let’s see what different storage class it comes with

kubectl get storageclass

kubectl describe storageclass standard

• Now, let’s just create a pod with MongoDB image and create PVC with standard class with access mode as ReadWriteOnce(RWO).

# example-rwo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dynamic-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 1Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rsvp-db
spec:
  replicas: 1
  selector:
    matchLabels:
      appdb: rsvpdb
  template:
    metadata:
      labels:
        appdb: rsvpdb
    spec:
      volumes:
        - name: voldb
          persistentVolumeClaim:
           claimName: dynamic-claim
      containers:
      - name: rsvpd-db
        image: teamcloudyuga/mongo:3.3
        volumeMounts:
        - name: voldb
          mountPath: /data/db
        ports:
        - containerPort: 27017

kubectl apply -f example-rwo.yaml

kubectl get pods

kubectl get pvc

kubectl describe pvc dynamic-claim

• Now, let’s just try to create a PV with access mode as ReadWriteMany(RWX) other than what the default storage class offers.

# example-rwx.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dynamic-claim-rwx
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: standard
  resources:
    requests:
      storage: 1Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rsvp-db1
spec:
  replicas: 2
  selector:
    matchLabels:
      appdb: rsvpdb
  template:
    metadata:
      labels:
        appdb: rsvpdb
    spec:
      volumes:
        - name: voldb1
          persistentVolumeClaim:
           claimName: dynamic-claim-rwx
      containers:
      - name: rsvpd-db1
        image: teamcloudyuga/mongo:3.3
        volumeMounts:
        - name: voldb1
          mountPath: /data/db
        ports:
        - containerPort: 27017

kubectl apply -f example-rwx.yaml

kubectl get pods

kubectl get pvc

It will make the pod and PVC be in pending status as the standard storage class doesn’t support ReadWriteMany(RWX) access mode as it comes only with one access mode ReadWriteOnce(RWO).

kubectl describe pvc dynamic-claim-rwx

Set-up RWX access-mode through NFS

As this standard storage class doesn’t have other access modes but what if the user wants to have the same persistent volume to be mounted in all the nodes of the kind cluster (want access mode as ReadWriteMany) so this can be configured in the kind cluster through NFS(Network File System).

To use NFS, one has to use an external provisioner to create a storage class for NFS as kubernetes doesn’t come with an internal provisioner for the same.

Some of the external provisioners are

• NFS Ganesha server and external provisioner
• NFS subdir external provisioner

For now, we will be using the NFS Ganesha server and external provisioner to configure an NFS storage class in the kind cluster.

The above image gives an overview of the NFS Ganesha server working where it will create a pod along with respective services and RBAC roles. The pod will use a standard storage class to provide persistence and then with the help of this an NFS storage class example-nfs will be created which will provide RWX access mode to be used in PVC.

• For this create an nfs-provisioner deployment with service and service account

kubectl apply -f deployment.yaml

The pods of nfs-provisioner can be checked by

kubectl get pods

• Also, provide different RBAC roles to this NFS-provisioner

kubectl apply -f rbac.yaml

• After this configure an NFS storage class

# class.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: example-nfs
provisioner: example.com/nfs
mountOptions:
  - vers=4.1

kubectl apply -f class.yaml

The storage class can be checked by

kubectl get storageclass

Here, one thing is to be observed in both storage classes that volumeBindingMode are different. This field controls the volume binding and dynamic provisioning.

As standard storage class has volumeBindingMode as WaitForFirstConsumer mode which will delay the binding and provisioning of a PersistentVolume until a Pod using the PersistentVolumeClaim is created.

On the other hand, nfs storage class  has volumeBindingMode as Immediate mode (default mode) which indicates that volume binding and dynamic provisioning occurs once the PersistentVolumeClaim is created.

• Then create the Persistent Volume Claim(PVC) for the NFS

# claim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs
spec:
  storageClassName: example-nfs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi

kubectl apply -f claim.yaml

Now, the Persistent Volume(PV) and Persistent Volume Claim(PVC) can be checked by

kubectl get pv,pvc

As NFS has been configured with RWX access-mode and if we again create the rsvp-db1 deployment (Pending statusas of now) with NFS as its storage class in Persistent Volume Claim then the deployment will be created along with Persistent Volume.

Conclusion

In this hands-on lab, we have seen how to configure RWX access-mode storage class in a kubernetes cluster.