ETCD in Kubernetes

ETCD in Kubernetes

12 December 2021
kubernetes
key-value
database
Etcd

To learn how ETCD works in Kubernetes

In the previous hands-on lab, we have seen about etcd, its installation with its implementation, and how to set up a three-node etcd local cluster.

Now, we will be exploring the working of etcd in Kubernetes and how it stores the information about the cluster, and how we can interact with etcd to fetch the cluster-info.

As we know, etcd is an open-source distributed key-value store that serves as a backbone of distributed systems. It is written in Go and uses the Raft consensus algorithm to manage a highly available cluster.

NOTE: Click on the LAB SETUP button on the right-hand side to get the lab-ready for performing hands-on. This will set up a terminal and an IDE to perform the lab.

Kubernetes and Etcd

As etcd is the backbone of the distributed systems, so projects like Kubernetes highly rely on etcd and use etcd as its primary data store. It is one of the critical components of a Kubernetes cluster and works as a brain of a cluster system as it holds the cluster state information.

In Kubernetes, the component that interacts with etcd is the API server which uses etcd's watch API to  monitor the cluster and store cluster-wide object configurations. etcd is also used to monitor changes to either the actual or the desired state of its system.

For a production-level Kubernetes cluster, we generally follow multi-master node configuration topology. In this kind of cluster, there can be two ways of deploying etcd.


Figure 1:Stacked ETCD Cluster
Figure 1:Stacked ETCD Cluster
  • Another is external etcd topology which is used to create additionally secure and resilient external etcd cluster
Figure 2:External ETCD Cluster
Figure 2:External ETCD Cluster

etcd is the component that can be present either as a system daemon service or as a pod. When kubeadm/minikube is used to install the Kubernetes cluster it is present as a pod in the kube-system namespace. 

kubectl get pods -n kube-system

This gives a list of pods in the kube-system namespace with the etcd-master name for a pod of etcd.

Configuration of etcd in Kubernetes

To know the configuration details of etcd that is running as a pod inside Kubernetes one can use the following command
kubectl describe pod etcd-master -n kube-system

This shows the details of etcd like IP address, annotations, labels, the path to certificates for interaction with etcd and much more.

Interacting with etcd in Kubernetes

  • To retrieve all the keys present in the etcd-master pod 
kubectl exec etcd-master -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt  --key /etc/kubernetes/pki/etcd/server.key --cert  /etc/kubernetes/pki/etcd/server.crt  get / --prefix --keys-only" 

This command helps to interact with the etcd-master pod through kubectl exec and ETCDCTL_API is the API version through which we want to interact with etcd  --cacert, --key and --cert is for TLS certificates that we will get from executing the describe command present above and get / --prefix --keys-only will give all the keys present in etcd.

  • To retrieve one of the key-value pairs from the list of keys got from the above command execution.
kubectl exec etcd-master -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt  --key /etc/kubernetes/pki/etcd/server.key --cert  /etc/kubernetes/pki/etcd/server.crt get /registry/secrets/kube-system/job-controller-token-<ID>"

Replace <ID>  with the ID from the above list of keys for registry/secrets/kube-system/job-controller-token-<ID>  key.

This command will give detailed information about the job-controller-token secret present in kube-system namespace.

  • Create a configmap my-config and retrieve it via an etcd pod.
kubectl create configmap my-config --from-literal=color=blue
kubectl exec etcd-master -n kube-system -- sh -c "ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/server.key --cert /etc/kubernetes/pki/etcd/server.crt get /registry/configmaps/default/my-config"

This command is similar to the earlier get command but the only thing that changed is to retrieve the created configmap from etcd.

Backup and Restore of etcd in Kubernetes

Earlier we have interacted with etcd pod by using kubectl exec but now to take backup and restore of the etcd pod first install etcdctl command-line tool for interaction which can be installed on Ubuntu using  

sudo apt install etcd-client
  • To take the backup we need to execute the following commands 
sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /tmp/etcd-backup.db

Here to take the backup, a snapshot (backup) file /tmp/etcd-backup.db is used and one can specify another path for snapshot also but make sure that file is not been used by etcd beforehand.

Now, sometimes it gets important to take the backup of etcd when working with remote etcd as it is the brain of Kubernetes cluster or at that time when one is having its CKA/CKAD exam.
  • To restore the backup etcd
sudo ETCDCTL_API=3 etcdctl --data-dir=/var/lib/etcd-backup snapshot restore /tmp/etcd-backup.db

This will store the backup etcd at /var/lib/etcd-backup location.

Conclusion

In this hands-on lab, we have learnt about how etcd in Kubernetes work, interacted with it, and saw how to take a backup of etcd and restore it.

How likely are you going to recommend this lab to your friends or colleagues?

Unlikely
Likely

Leave a comment:

About the Author

Oshi Gupta

Oshi Gupta

DevOps Engineer & Technical Writer, CloudYuga

Oshi Gupta is currently working as DevOps Engineer and Technical Writer at CloudYuga. She is working on Kubernetes and different cloud-native technologies.