In the earlier hands-on lab, we saw how etcd works with Kubernetes and how to access etcd. Now we will be learning how etcd stores secrets and can one can access those secrets or not?

As we know containerized applications running in Kubernetes almost always need some access to external resources like secrets, passwords, keys, or tokens to gain access. Kubernetes Secrets lets you securely store these items, by removing the need to store them in pod definitions or container images.

All these secrets are stored in etcd which is the brain of Kubernetes. But does etcd keep them encrypted as these secrets are sensitive and not to be shared with everyone?

The answer is NO as by default Kubernetes only encode them by using base64 encoding which stores these secrets in plaintext(ASCII format) and doesn’t encrypt them which makes them insecure.

Create a Secret and Decode it

• Create a secret with a name first and provide key-value through --from-literal flag

kubectl create secret generic first --from-literal=foo=bar

kubectl get secrets

kubectl describe secret first

As on executing the above command, we will not be able to see the key's value but only its size.

kubectl get secret first -o yaml 

This command will show the key's value in an encoded format and its original value can be decoded easily by applying the below command.

echo <ENCODED_VALUE>| base64 --decode

where replace the <ENCODED_VALUE> with key's encoded value.

How etcd stores secrets?

•  Retrieve the details of etcd pod from kube-system namespace

kubectl get pods -n kube-system

kubectl describe pod etcd-master -n kube-system

 Retrieve the secret first created earlier via etcd pod.

kubectl -n kube-system exec -it etcd-master -- sh -c "ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key etcdctl --endpoints=https://127.0.0.1:2379 get /registry/secrets/default/first"

This will give both, the key and its value in a readable form and one who has access to etcd can easily retrieve these secrets which contains sensitive information. So as of now, we accessed secrets inside etcd as Kubernetes-admin but how to give access to non-Kubernetes admin. This can be done with the help of RBAC.

Allowing User-Group to access secrets in ETCD

As far we have seen how to create Kubernetes secrets, how base64 works on it and to access them via etcd. Now we will be allowing a certain group of users to access secrets via RBAC ClusterRole and ClusterRoleBinding concept other than Kubernetes-admin.

• Create user user1 keys through OpenSSL which will set up the certificates for the user and then set credentials for it through kubectl config and then give them access to secrets through cluster-role.

openssl genrsa -out user1.key 2048

openssl req -new -key user1.key -out user1.csr -subj "/CN=user1/O=developer"

openssl x509 -req -in user1.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out user1.crt -days 500

kubectl config set-credentials user1 --client-certificate=user1.crt --client-key=user1.key

kubectl config view

• After the above process where after generating keys and setting credentials for the user, a cluster role will be created for the user that will give access to secrets and perform operations on them.

From the above diagram we can see, a group developer is made to access secrets other than k8s-admin with the help of cluster role binding and on that secret, various operations can be performed like get, delete, list, etc through cluster role.

kubectl apply -f cluster-role-secret-reader.yaml

• Then bind the cluster-role via cluster-role binding 

kubectl apply -f cluster-binding-secret-reader.yaml

And if this user user1 from group developer has access to secrets then the user can easily access the secrets inside etcd also so one has to think beforehand giving access to secrets to the right persons or a group of persons (non - Kubernetes admin) as these secrets hold crucial pieces of information.

Conclusion

In this hands-on lab, we saw how to create Kubernetes secrets, decode them, and how to give permissions to users or groups who can access secrets in etcd which makes secrets insecure.