Bitnami’s SealedSecrets – Hands-On Blog

Managing Kubernetes Secrets externally using the Bitnami’s SealedSecrets.

Nowadays we are using GitOps for application deployment and for that we tend to put all the application’s information and configuration on Git, but can we do the same with Kubernetes Secrets YAML file on Git? The answer is definitely NO, as the Secret’s file contains just the base64 encoded value of our sensitive information; which can be passwords, keys, certificates, tokens etc. 

Following is a sample YAML file for a Secret:-

apiVersion: v1
kind: Secret
metadata: 
  name: mysecret
  type: Opaque
  data: 
    username: Y2xvdWR5dWdh 
    password: Y2xvdWR5dWdhMTIz

And we can easily decode the sensitive information with base64 command like following:-

echo -n Y2xvdWR5dWdh | base64 -d

So, definitely we should NOT store Kubernetes Secrets on Git.  There are many different ways to externalize k8s secrets like Hashicorp’s Vault, Helm Secrets, Bitnami’s SealedSecret etc. In this blog we are going to explore Bitnami’s Sealed Secret. 

Following is an example of the SealedSecret :-

kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: default
spec:
  encryptedData:
    password: AgA+Bhm+aD7uwSEau0Ax+f7s9QSiKdYWaKXiIRWjE1PPwMerGzu6WrX5RqPjBxPxQ/uh/0fpDlbK3LIyuyY+atRr464FryG+oWmRBPTR+JBNReyjumIu49CHXqjOPa1x9//oPFWIYwnQWuFDwU8NH4Ghv9qtxnbcFdeLyr4OOy3LDpgnGcemSuPVqA/Fuzw+XzcyQnfpPv5ZhWr4mr/H6U9rNTwwLxhXh83UAtVceLaezflLVFQ5Dz4ki/MrvgD+ZWPp4UsR7inllZFVR9fKA9Xlv4cIzZvXgjw1SeC3Ey43fwK5SwpKpqGOAGpjWl9saHkswbXdGVPlnMuRXYEPibjEjreJUodtZm8+A1ipC1/dyxAjo8J9yvZu4+cCQ0pV36hkwoQm4hPB8Junv79W2xwLxWZxgUUDdSNogVXeFbxtZWMdmfbHOgMpg9TVNRq4sJjNVl+ea5D/d7bqSCIR8EBfA7/nGCINMMa2gOpTwWw0qxDqc2qPc4qpr4gm+8dtB+hNlN/SsQVo5eVloJQOBNmnQc6LicUIpAonQwCY/wskR2BKjG/NewD9ius2rA5mycYcNERY/vqpGxt/9QioVKNaKcusB61fXLv5dLxKpyNoW072s0ui6nJ5qdfuevbHIwg4jmNz/bC/nQUihvIny6kVXsOZHqn6/TqSbihc+kOnqlrdutbX8M3YEYjdqdfkzWRxXb7AsVIWrBbkWz0=
    username: AgByt2AUBOWtd8kS8Fd84F/TeUrPi3Vy3E3RT2+ToKG2NPx0AofjvW6j0jeJDjy9wBrSwQIfexcQe+HXtbc5Jnj0hMNpdNqZSwvNniwRATZmV/kUYHuPkVr5b5tOARE2TWQodQim5sgY+3itsYs6+8F9A+R0NZT69VKrXRX2HHBdxRwZBQhoocKiTrTXF4JpfzY+eYPHYM4zpx3ifWIYkCiSln2U5m1sEJpZMAsXDPb3lt7mjm6N8ECBtESLAAuJOkcfCfE9voBkUetOriLx3Im7njCSxZGe8CqkrxZBBygQrjghs8rk42OCy+A9zCCOH3zqZ0cIqdJNMp3UnN68V7RJFAs6Y2kEQUHz4KKBSXYosiH4i87nI4vv4aUxeiPjeWyTNEEvzCsK1eBuVn0eYEhBhUanllmuUfPfxX0efOrWd7JO44wtTUmLnJI0f8JfFWsbsWcscnQOSjynX6n1HdNJLDuCZ3d/21oInkXAesB1umv+XOyFlbmmMmwhypNHoJy02QtxiJ6ZrxV2vnBulSl10DBw/8qHdq6UmhM24uHGbrEhbVV1jUekBOhsZ/wA19B+4dhNhDDuCtQUThuZyg0AnNN/YULVvC+0QHYazR7edNirmmEFQlEYF/n1izOPxXJFIpgMu9a60GYm6UWVnJyPeblzFFbCpYZfZNN45Zxd0ee0gMnYrW/TKwKdOXIKpC1J+Oe4f/QRO8I=
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: mysecret
      namespace: default
    type: Opaque

which can be stored anywhere. The SealedSecret Custom Resource seals the Kubernetes Secret object, which can we unsealed only with the private key, stored in the given k8s cluster. The SealedSecret object has a template section which encodes all the fields we want the to put in the unsealed Secret, including the type of it.

Bitnami’s SealedSecret has two components, a cluster-side Controller and a client-side utility called kubeseal.At a high level we’ll do following:-

• Have a Kubernetes Secret Configuration File 
• Use kubeseal  program to create a respective SealedSecret
• Share SealedSecret anywhere
• Apply the SealedSecret to the Kubernetes Cluster, which has the respective Controller running
• The Controller unseals the SealedSecret to create Kubernetes Secret

Let us now explore in detail.

Cluster-side Controller/Operator

The Controller is responsible to decrypt the the SealedSecret and the respective Kubernetes Secret Object.  We can install the controller using following command :-

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml

This would install the controller into the kube-system namespace and create the resources called SealedSecret. This would also setup respective service-account and necessary RBAC roles.

kubectl get pods -n kube-system

kubectl explain sealedsecret

The installation would also create a Public/Private key pair. The Public Key can be given to anyone to encrypt the data (SealedSecret), which can only be decrypted by the private key; available only with the controller.  

The Public Key is used by the Client-side program kubeseal to create the SealedSecret, which we’ll look at next. 

Client-side utility called Kubeseal

kubeseal is a CLI tool that uses the Public Key generated by the Controller, which we discussed in previous section. kubeseal gets the Public Key via the Kubernetes APIServer or it can be downloaded and shared manually as well. We can install the kubeseal on Linux, using the following commands:-

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/kubeseal-linux-amd64 -O kubeseal

sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Once kubeseal is installed we can take a Kubernetes Secret and generate SealedSecret. 

apiVersion: v1
kind: Secret
metadata: 
  name: mysecret
  type: Opaque
  data: 
    username: Y2xvdWR5dWdh 
    password: Y2xvdWR5dWdhMTIz

cat secret.yaml

kubeseal --format=yaml < secret.yaml > sealed-secret.yaml

The above would take take a Kubernetes Secret as input and spit out a SealedSecret in the sealed-secret.yaml.

cat sealed-secret.yaml

This sealed-secret.yaml file now can be taken anywhere and stored publicly. You can even share on Twitter. Suppose this is stored on the Git for our GitOps workflow then it would be decrypted/unsealed by the the SealedSecret Controller at runtime; as we apply the manifests to the right cluster.  

Decryption using Cluster-side Controller

Let us now apply the Sealedsecret YAML file, which we created in the last step. 

kubectl apply -f sealed-secret.yaml

Above command would reach out  to the Sealedsecret Controller installed in the cluster and create K8s Secret in the respective namespace. 

kubectl get secret -n default 

kubectl get secret mysecret -o yaml -n default

Sealed Secrets Scopes

K8s secrets are namespaced objects and when we create the SealedSecret object we need to somehow preserve that information. It should not happen that some un-authorized/malicious user renames it or apply it another namespace.  But sometimes we may be need to rename or deploy in a different namespace as well. 

To accommodate above requirements, SealedSecrets support following scopes.

Strict

This is the default type, in which secret is sealed with exactly the same name and namespace. These attributes become part of the encrypted data and can not changed while decrypting, to create the secret object. 

Namespace-wide

We can rename the secret only within the given namespace. 

Cluster-wide

We can unseal the in any namespace and can be given any name of our choice. 

We can define the scope using  --scope flag in kubeseal, like following :-

kubeseal --scope cluster-wide <secret.yaml >sealed-secret.yaml

We can also use annotations within the k8s Secret to apply scopes before passing the configuration to kubeseal 

sealedsecrets.bitnami.com/namespace-wide: "true" -> for namespace-wide<a sealedsecrets.bitnami.com/cluster-wide: "true" -> for cluster-wide

If we don’t specify the scope flag, then kubeseal will automatically use the Strict Scope by default.

Secret Renewal and Rotation

As we install the SealedSecret Controller on the Kubernetes cluster, it creates a public/private key combination to encrypt and decrypt the actual users secrets. For security reason, this key combination gets renewed every 30 days and it can be customised as well. Please take a note that this is not a rotation, in which we replace the old ones with the new ones. 

SealedSecret Controller keeps track of the older private keys to decrypt any users secrets created earlier. Only the latest public key would be used to encrypt users secrets. 

For actual users secrets it is always a good practice to rotate them to create new SealedSecrets. 

For more details on above please checkout the documentation. 

So in this blog we have seen how to use Bitnami's SealedSecret to store secrets publicly, which can use very useful in scenarios like GitOps. Feel free to reach out, if you have any questions.