Kubernetes Secrets are not encrypted by default. Only encoded base64

Is Kubernetes Secrets…. A Secret?

To know how one can get access to Kubernetes secrets

5 May 2022
secrets
Etcd
base64
encoding
kubernetes

In the earlier hands-on lab, we saw how etcd works with Kubernetes and how to access etcd. Now we will be learning how etcd stores secrets and can one can access those secrets or not?

As we know containerized applications running in Kubernetes almost always need some access to external resources like secrets, passwords, keys, or tokens to gain access. Kubernetes Secrets lets you securely store these items, by removing the need to store them in pod definitions or container images.

All these secrets are stored in etcd which is the brain of Kubernetes. But does etcd keep them encrypted as these secrets are sensitive and not to be shared with everyone?

The answer is NO as by default Kubernetes only encode them by using base64 encoding which stores these secrets in plaintext(ASCII format) and doesn’t encrypt them which makes them insecure.

Figure 1: Creation of Secret
Figure 1: Creation of Secret

Create a Secret and Decode it

  • Create a secret with a name first and provide key-value through --from-literal flag
kubectl create secret generic first --from-literal=foo=bar
kubectl get secrets
kubectl describe secret first

As on executing the above command, we will not be able to see the key's value but only its size.

kubectl get secret first -o yaml 

This command will show the key's value in an encoded format and its original value can be decoded easily by applying the below command.

echo <ENCODED_VALUE>| base64 --decode

where replace the <ENCODED_VALUE> with key's encoded value.

On executing the above commands we can infer that secrets in Kubernetes are not secure and one can easily access them.

How etcd stores secrets?

  •  Retrieve the details of etcd pod from kube-system namespace
kubectl get pods -n kube-system
kubectl describe pod etcd-master -n kube-system

 Retrieve the secret first created earlier via etcd pod.

kubectl -n kube-system exec -it etcd-master -- sh -c "ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key etcdctl --endpoints=https://127.0.0.1:2379 get /registry/secrets/default/first"

This will give both, the key and its value in a readable form and one who has access to etcd can easily retrieve these secrets which contains sensitive information. So as of now, we accessed secrets inside etcd as Kubernetes-admin but how to give access to non-Kubernetes admin. This can be done with the help of RBAC.

Allowing User-Group to access secrets in ETCD

As far we have seen how to create Kubernetes secrets, how base64 works on it and to access them via etcd. Now we will be allowing a certain group of users to access secrets via RBAC ClusterRole and ClusterRoleBinding concept other than Kubernetes-admin.

  • Create user user1 keys through OpenSSL which will set up the certificates for the user and then set credentials for it through kubectl config and then give them access to secrets through cluster-role.
openssl genrsa -out user1.key 2048
openssl req -new -key user1.key -out user1.csr -subj "/CN=user1/O=developer"
openssl x509 -req -in user1.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out user1.crt -days 500
kubectl config set-credentials user1 --client-certificate=user1.crt --client-key=user1.key
kubectl config view
  • After the above process where after generating keys and setting credentials for the user, a cluster role will be created for the user that will give access to secrets and perform operations on them.
Figure 2: RBAC ClusterRole and ClusterRoleBinding
Figure 2: RBAC ClusterRole and ClusterRoleBinding

From the above diagram we can see, a group developer is made to access secrets other than k8s-admin with the help of cluster role binding and on that secret, various operations can be performed like get, delete, list, etc through cluster role.

kubectl apply -f cluster-role-secret-reader.yaml
  • Then bind the cluster-role via cluster-role binding 
kubectl apply -f cluster-binding-secret-reader.yaml

And if this user user1 from group developer has access to secrets then the user can easily access the secrets inside etcd also so one has to think beforehand giving access to secrets to the right persons or a group of persons (non - Kubernetes admin) as these secrets hold crucial pieces of information.

Conclusion

In this hands-on lab, we saw how to create Kubernetes secrets, decode them, and how to give permissions to users or groups who can access secrets in etcd which makes secrets insecure.

How likely are you going to recommend this lab to your friends or colleagues?

Unlikely
Likely

Leave a comment:

About the Author

Oshi Gupta

Oshi Gupta

DevOps Engineer & Technical Writer, CloudYuga

Oshi Gupta works as a DevOps Engineer and Technical Writer at CloudYuga Technologies. She is a CKA certified and has been selected for LFX mentorship in Spring 2022 for CNCF Kyverno. She loves writing blogs and is keen to learn about various cloud-native technologies. Besides this, she loves cooking, badminton, traveling, and yoga.