In the earlier hands-on lab, we saw how etcd works with Kubernetes and how to access etcd. Now we will be learning how etcd stores secrets and can one can access those secrets or not?
As we know containerized applications running in Kubernetes almost always need some access to external resources like secrets, passwords, keys, or tokens to gain access. Kubernetes Secrets lets you securely store these items, by removing the need to store them in pod definitions or container images.
All these secrets are stored in etcd which is the brain of Kubernetes. But does etcd keep them encrypted as these secrets are sensitive and not to be shared with everyone?
The answer is NO as by default Kubernetes only encode them by using base64 encoding which stores these secrets in plaintext(ASCII format) and doesn’t encrypt them which makes them insecure.
Create a Secret and Decode it
- Create a secret with a name
firstand provide key-value through
kubectl create secret generic first --from-literal=foo=bar
kubectl get secrets
kubectl describe secret first
As on executing the above command, we will not be able to see the key's value but only its size.
kubectl get secret first -o yaml
This command will show the key's value in an encoded format and its original value can be decoded easily by applying the below command.
echo <ENCODED_VALUE>| base64 --decode
where replace the
<ENCODED_VALUE> with key's encoded value.
How etcd stores secrets?
- Retrieve the details of etcd pod from
kubectl get pods -n kube-system
kubectl describe pod etcd-master -n kube-system
Retrieve the secret
first created earlier via etcd pod.
kubectl -n kube-system exec -it etcd-master -- sh -c "ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key etcdctl --endpoints=https://127.0.0.1:2379 get /registry/secrets/default/first"
This will give both, the key and its value in a readable form and one who has access to etcd can easily retrieve these secrets which contains sensitive information. So as of now, we accessed secrets inside etcd as Kubernetes-admin but how to give access to non-Kubernetes admin. This can be done with the help of RBAC.
Allowing User-Group to access secrets in ETCD
As far we have seen how to create Kubernetes secrets, how base64 works on it and to access them via etcd. Now we will be allowing a certain group of users to access secrets via RBAC ClusterRole and ClusterRoleBinding concept other than Kubernetes-admin.
- Create user
user1keys through OpenSSL which will set up the certificates for the user and then set credentials for it through
kubectl configand then give them access to secrets through cluster-role.
openssl genrsa -out user1.key 2048
openssl req -new -key user1.key -out user1.csr -subj "/CN=user1/O=developer"
openssl x509 -req -in user1.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out user1.crt -days 500
kubectl config set-credentials user1 --client-certificate=user1.crt --client-key=user1.key
kubectl config view
- After the above process where after generating keys and setting credentials for the user, a cluster role will be created for the user that will give access to secrets and perform operations on them.
From the above diagram we can see, a
group developer is made to access secrets other than k8s-admin with the help of cluster role binding and on that
secret, various operations can be performed like
get, delete, list, etc through cluster role.
kubectl apply -f cluster-role-secret-reader.yaml
- Then bind the cluster-role via cluster-role binding
kubectl apply -f cluster-binding-secret-reader.yaml
And if this user
group developer has access to secrets then the user can easily access the secrets inside etcd also so one has to think beforehand giving access to secrets to the right persons or a group of persons (non - Kubernetes admin) as these secrets hold crucial pieces of information.
In this hands-on lab, we saw how to create Kubernetes secrets, decode them, and how to give permissions to users or groups who can access secrets in etcd which makes secrets insecure.