Nowadays we are using GitOps for application deployment and for that we tend to put all the application's information and configuration on Git, but can we do the same with
Kubernetes Secrets YAML file
on Git? The answer is definitely
NO, as the Secret's file contains just the base64 encoded value of our sensitive information; which can be
passwords, keys, certificates, tokens etc.
Following is a sample YAML file for a Secret:-
And we can easily decode the sensitive information with
base64 command like following:-
echo -n Y2xvdWR5dWdh | base64 -d
So, definitely we should NOT store Kubernetes Secrets on Git. There are many different ways to externalize k8s secrets like Hashicorp's Vault, Helm Secrets, Bitnami's SealedSecret etc. In this blog we are going to explore Bitnami's Sealed Secret.
Following is an example of the
which can be stored anywhere. The
Kubernetes Secret object, which can we unsealed only with the private key, stored in the given k8s cluster. The
object has a
section which encodes all the fields we want the to put in the unsealed
Secret, including the type of it.
Bitnami's SealedSecret has two components, a cluster-side Controller and a client-side utility called
kubeseal.At a high level we'll do following:-
- Have a Kubernetes Secret Configuration File
kubesealprogram to create a respective
- Apply the
SealedSecretto the Kubernetes Cluster, which has the respective
- The Controller unseals the
SealedSecretto create Kubernetes Secret
Let us now explore in detail.
The Controller is responsible to decrypt the the
SealedSecret and the respective
Kubernetes Secret Object. We can install the controller using following command :-
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml
This would install the controller into the
kube-system namespace and create the resources called
SealedSecret. This would also setup respective service-account and necessary RBAC roles.
kubectl get pods -n kube-system
kubectl explain sealedsecret
The installation would also create a
Public/Private key pair. The Public Key can be given to anyone to encrypt the data (SealedSecret), which can only be decrypted by the private key; available only with the controller.
Public Key is used by the Client-side program
kubeseal to create the
SealedSecret, which we'll look at next.
Client-side utility called Kubeseal
kubeseal is a CLI tool that uses the
Public Key generated by the
Controller, which we discussed in previous section.
kubeseal gets the Public Key via the
Kubernetes APIServer or it can be downloaded and shared manually as well. We can install the
kubeseal on Linux, using the following commands:-
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/kubeseal-linux-amd64 -O kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal
kubeseal is installed we can take a
Kubernetes Secret and generate
kubeseal --format=yaml < secret.yaml > sealed-secret.yaml
The above would take take a
Kubernetes Secret as input and spit out a
SealedSecret in the
sealed-secret.yaml file now can be taken anywhere and stored publicly. You can even share on Twitter. Suppose this is stored on the Git for our GitOps workflow then it would be decrypted/unsealed by the the
SealedSecret Controller at runtime; as we apply the manifests to the right cluster.
Decryption using Cluster-side Controller
Let us now apply the
Sealedsecret YAML file, which we created in the last step.
kubectl apply -f sealed-secret.yaml
Above command would reach out to the
Sealedsecret Controller installed in the cluster and create K8s Secret in the respective namespace.
kubectl get secret -n default
kubectl get secret mysecret -o yaml -n default
Sealed Secrets Scopes
K8s secrets are namespaced objects and when we create the SealedSecret object we need to somehow preserve that information. It should not happen that some un-authorized/malicious user renames it or apply it another namespace. But sometimes we may be need to rename or deploy in a different namespace as well.
To accommodate above requirements, SealedSecrets support following scopes.
This is the default type, in which secret is sealed with exactly the same name and namespace. These attributes become part of the encrypted data and can not changed while decrypting, to create the secret object.
We can rename the secret only within the given namespace.
We can unseal the in any namespace and can be given any name of our choice.
We can define the scope using
--scope flag in
kubeseal, like following :-
kubeseal --scope cluster-wide <secret.yaml >sealed-secret.yaml
We can also use annotations within the
k8s Secret to apply scopes before passing the configuration to kubeseal
If we don't specify the
scope flag, then
kubeseal will automatically use the
Strict Scope by default.
Secret Renewal and Rotation
As we install the
SealedSecret Controller on the Kubernetes cluster, it creates a
public/private key combination to
encrypt and decrypt the actual users secrets. For security reason, this key combination gets
renewed every 30 days and it can be customised as well. Please take a note that this is
not a rotation, in which we replace the old ones with the new ones.
SealedSecret Controller keeps
track of the older private keys to
users secrets created earlier. Only the
latest public key would be used to encrypt users secrets.
actual users secrets it is always a good practice to rotate them to create
For more details on above please checkout the documentation.
So in this blog we have seen how to use
Bitnami's SealedSecret to store secrets publicly, which can use very useful in scenarios like GitOps. Feel free to reach out, if you have any questions.