Is Kubernetes Secrets…. A Secret?

5 May 2022
encoding
base64
kubernetes
secrets
Etcd

To know how one can get access to Kubernetes secrets

In the earlier hands-on lab, we saw how etcd works with Kubernetes and how to access etcd. Now we will be learning how etcd stores secrets and can one can access those secrets or not?

As we know containerized applications running in Kubernetes almost always need some access to external resources like secrets, passwords, keys, or tokens to gain access. Kubernetes Secrets lets you securely store these items, by removing the need to store them in pod definitions or container images.

All these secrets are stored in etcd which is the brain of Kubernetes. But does etcd keep them encrypted as these secrets are sensitive and not to be shared with everyone?

The answer is NO as by default Kubernetes only encode them by using base64 encoding which stores these secrets in plaintext(ASCII format) and doesn’t encrypt them which makes them insecure.

Figure 1: Creation of Secret
Figure 1: Creation of Secret

Create a Secret and Decode it

  • Create a secret with a name first and provide key-value through --from-literal flag
kubectl create secret generic first --from-literal=foo=bar
kubectl get secrets
kubectl describe secret first

As on executing the above command, we will not be able to see the key's value but only its size.

kubectl get secret first -o yaml 

This command will show the key's value in an encoded format and its original value can be decoded easily by applying the below command.

echo <ENCODED_VALUE>| base64 --decode

where replace the <ENCODED_VALUE> with key's encoded value.

On executing the above commands we can infer that secrets in Kubernetes are not secure and one can easily access them.

How etcd stores secrets?

  •  Retrieve the details of etcd pod from kube-system namespace
kubectl get pods -n kube-system
kubectl describe pod etcd-master -n kube-system

 Retrieve the secret first created earlier via etcd pod.

kubectl -n kube-system exec -it etcd-master -- sh -c "ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key etcdctl --endpoints=https://127.0.0.1:2379 get /registry/secrets/default/first"

This will give both, the key and its value in a readable form and one who has access to etcd can easily retrieve these secrets which contains sensitive information. So as of now, we accessed secrets inside etcd as Kubernetes-admin but how to give access to non-Kubernetes admin. This can be done with the help of RBAC.

Allowing User-Group to access secrets in ETCD

As far we have seen how to create Kubernetes secrets, how base64 works on it and to access them via etcd. Now we will be allowing a certain group of users to access secrets via RBAC ClusterRole and ClusterRoleBinding concept other than Kubernetes-admin.

  • Create user user1 keys through OpenSSL which will set up the certificates for the user and then set credentials for it through kubectl config and then give them access to secrets through cluster-role.
openssl genrsa -out user1.key 2048
openssl req -new -key user1.key -out user1.csr -subj "/CN=user1/O=developer"
openssl x509 -req -in user1.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out user1.crt -days 500
kubectl config set-credentials user1 --client-certificate=user1.crt --client-key=user1.key
kubectl config view
  • After the above process where after generating keys and setting credentials for the user, a cluster role will be created for the user that will give access to secrets and perform operations on them.
Figure 2: RBAC ClusterRole and ClusterRoleBinding
Figure 2: RBAC ClusterRole and ClusterRoleBinding

From the above diagram we can see, a group developer is made to access secrets other than k8s-admin with the help of cluster role binding and on that secret, various operations can be performed like get, delete, list, etc through cluster role.

Copy Code
Loading...
kubectl apply -f cluster-role-secret-reader.yaml
  • Then bind the cluster-role via cluster-role binding 
Copy Code
Loading...
kubectl apply -f cluster-binding-secret-reader.yaml

And if this user user1 from group developer has access to secrets then the user can easily access the secrets inside etcd also so one has to think beforehand giving access to secrets to the right persons or a group of persons (non - Kubernetes admin) as these secrets hold crucial pieces of information.

Conclusion

In this hands-on lab, we saw how to create Kubernetes secrets, decode them, and how to give permissions to users or groups who can access secrets in etcd which makes secrets insecure.

About the Author

Oshi Gupta

Oshi Gupta

DevOps Engineer & Technical Writer, CloudYuga

Oshi Gupta is a final year undergraduate student and currently working as an Intern at CloudYuga. She is working on Kubernetes and different cloud-native technologies. She also has been a student mentor for the Google Cloud Career Readiness program.