To learn how ETCD works in Kubernetes

In the previous blog, we have seen about etcd, its installation with its implementation, and how to set up a three-node etcd local cluster.

Now, we will be exploring the working of etcd in Kubernetes and how it stores the information about the cluster, and how we can interact with etcd to fetch the cluster-info.

As we know, etcd is an open-source distributed key-value store that serves as a backbone of distributed systems. It is written in Go and uses the Raft consensus algorithm to manage a highly available cluster.

Kubernetes and Etcd

As etcd is the backbone of the distributed systems, so projects like Kubernetes highly rely on etcd and use etcd as its primary data store. It is one of the critical components of a Kubernetes cluster and works as a brain of a cluster system as it holds the cluster state information.

In Kubernetes, the component that interacts with etcd is the API server which uses etcd’s watch API to monitor the cluster and store cluster-wide object configurations. etcd is also used to monitor changes to either the actual or the desired state of its system.

For a production-level Kubernetes cluster, we generally follow multi-master node configuration topology. In this kind of cluster, there can be two ways of deploying etcd.

• One is a stacked etcd topology where etcd is present on the master nodes itself as given below

• Another is external etcd topology which is used to create additionally secure and resilient external etcd cluster

etcd is the component that can be present either as a system daemon service or as a pod. When kubeadm/minikube is used to install the Kubernetes cluster it is present as a pod in the kube-system namespace. 

kubectl get pods -n kube-system

This gives a list of pods in the kube-system namespace with the etcd-master name for a pod of etcd.

Configuration of etcd in Kubernetes

To know the configuration details of etcd that is running as a pod inside Kubernetes one can use the following command

kubectl describe pod etcd-master -n kube-system

This shows the details of etcd like IP address, annotations, labels, the path to certificates for interaction with etcd and much more.

Interacting with etcd in Kubernetes

• To retrieve all the keys present in the etcd-master pod 

kubectl exec etcd-master -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt  --key /etc/kubernetes/pki/etcd/server.key --cert  /etc/kubernetes/pki/etcd/server.crt  get / --prefix --keys-only" 

This command helps to interact with the etcd-master pod through kubectl exec and ETCDCTL_API is the API version through which we want to interact with etcd  --cacert, --key and --cert is for TLS certificates that we will get from executing the describe command present above and get / --prefix --keys-only will give all the keys present in etcd.

• To retrieve one of the key-value pairs from the list of keys got from the above command execution.

kubectl exec etcd-master -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt  --key /etc/kubernetes/pki/etcd/server.key --cert  /etc/kubernetes/pki/etcd/server.crt get /registry/secrets/kube-system/job-controller-token-<ID>"

Replace <ID>  with the ID from the above list of keys for registry/secrets/kube-system/job-controller-token-<ID>  key.

This command will give detailed information about the job-controller-token secret present in kube-system namespace.

• Create a configmapmy-config and retrieve it via an etcd pod.

kubectl create configmap my-config --from-literal=color=blue

kubectl exec etcd-master -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/server.key --cert /etc/kubernetes/pki/etcd/server.crt get /registry/configmaps/default/my-config"

This command is similar to the earlier get command but the only thing that changed is to retrieve the created configmap from etcd.

Backup and Restore of etcd in Kubernetes

Earlier we have interacted with etcd pod by using kubectl exec but now to take backup and restore of the etcd pod first install etcdctl command-line tool for interaction which can be installed on Ubuntu using  

sudo apt install etcd-client

• To take the backup we need to execute the following commands 

sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /tmp/etcd-backup.db

Here to take the backup, a snapshot (backup) file /tmp/etcd-backup.db is used and one can specify another path for snapshot also but make sure that file is not been used by etcd beforehand.

Now, sometimes it gets important to take the backup of etcd when working with remote etcd as it is the brain of Kubernetes cluster or at that time when one is having its CKA/CKAD exam.

• To restore the backup etcd

sudo ETCDCTL_API=3 etcdctl --data-dir=/var/lib/etcd-backup snapshot restore /tmp/etcd-backup.db

This will store the backup etcd at /var/lib/etcd-backup location.

Conclusion

In this blog, we have learnt about how etcd in Kubernetes work, interacted with it, and saw how to take a backup of etcd and restore it.