To know how to inject secrets into the kubernetes pods via vault agent sidecar injector

Keeping Kubernetes secret a secret (secure) is one of the challenges an organization faces as these secrets are only encoded and not encrypted by default at REST. These secrets are stored in ETCD, a key/values store with all the Kubernetes configurations. Creating a Kubernetes secret and using it for in-cluster apps is a good option but using it for out-of-cluster apps is a bit risky as there are more chances of secrets getting compromised.

One of the most effective ways to store secrets in a secure manner is to use an external secret management system like Vault by integrating it with Kubernetes. This not only helps in storing secrets securely but also helps in rotating secrets and generating dynamic secrets as per the requirements.

In this hands-on lab, we will be going to see how to integrate Vault with Kubernetes and inject secrets in the pods through vault agent sidecar injectors.

As we are going to see how to inject secrets into Kubernetes pods via vault agent sidecar injector. So let’s break it into two parts and first understand about vault agent and then about injector and sidecar pattern.

Vault Agent

As Vault is a secret management tool and helps in securely storing the secrets. As Vault provides various methods for authentication to users and policies for authorization. When Vault is being used by applications, sometimes it becomes difficult to maintain applications that invoke Vault API for authentication and accessing secrets. So for this Vault Agent gets introduced which helps applications in adopting Vault in a much simpler way. 

Vault Agent is a client daemon(background process) that helps in easy authentication, caching of tokens and leases, and provides templates for retrieving secrets and dynamic secret generations.

In the above diagram, the vault agent helps authenticate to Vault by Auth methods, for example, the Kubernetes method which uses a service account token for authentication.As soon as the vault certifies it then it allows accessing secrets.

Injector

An injector is nothing but a Kubernetes mutation webhook controller which helps in intercepting the pod events and mutates it if the vault agent specific annotations exist in the resource’s spec attribute.

 We will be seeing it in much more detail in the later sections.

SideCar Pattern

In this pattern, generally along with application containers in a pod, a sidecar container is also introduced which enhances the functionality of the application without tight coupling. As the application and sidecar containers are both deployed on the same pod, they share the same resources as the local file system memory. More about the sidecar pattern can be found here.

As we have got an idea about vault agent, injector, and sidecar pattern, let’s now see how we can inject secrets into the Kubernetes pods via the vault agent sidecar injector. 

Working on Vault Agent Sidecar Injector for secrets in Pods

When the vault is installed via helm-charts in Kubernetes, it generates two pods. First is the vault pod which runs the vault server in development mode generally and the other is the vault-agent-injector pod which performs injection based on the annotations present.   

As we know that Vault agent injector is a controller that helps in intercepting pod events like create or update to check if any agent-specific annotation is applied to a pod and it will help in adding a vault sidecar and init container to the Kubernetes pods at runtime.

The init container will do the initial task of authenticating using the pod service account token and retrieving secrets from the vault server. The sidecar container is required for creating on-demand dynamic secrets with a lease on it.

Lab for injecting secrets through Vault Agent Sidecar Injector

 In the lab section, we will be going to see how to install a vault in Kubernetes through helm and then going to enable authentication in the vault for Kubernetes, and then going to inject secrets into the pods.

Install Vault Helm Chart

• Install Helm through the script

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

• Add the Hashicorp Helm repository and install the latest version of the Vault server in development mode.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm install vault hashicorp/vault --set "server.dev.enabled=true"

• Check the vault deployment and pods in the default namespace

kubectl get deploy,pods

The vault-0 pod makes the Vault server run in development mode and the vault-agent-injector pod performs the injection based on the annotations present.

Configure Kubernetes Authentication

• Connect to the vault server pod vault-0 and configure an app named policy which has read capabilities.

kubectl exec vault-0 -- ls

cat <<EOF > app-policy.hcl
path "secret*" {
  capabilities = ["read"]
}
EOF

kubectl cp app-policy.hcl default/vault-0:/home/vault

kubectl exec vault-0 -- vault policy write app /home/vault/app-policy.hcl

• Next, enable the Kubernetes Authentication method inside the vault-0 pod

kubectl exec vault-0 -- vault auth enable kubernetes

• Now, configure the Kubernetes Authentication method to access the location of Kubernetes API for verifying the service account token in the vault server.

kubectl exec vault-0 -it -- /bin/sh

vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

• Next, create a Kubernetes authentication role and attach a policy to vault server.

kubectl exec vault-0 -- vault write auth/kubernetes/role/myapp \
bound_service_account_names=app \
bound_service_account_namespaces=default \
policies=app \
ttl=1h

Create Secret in Vault

• Let’s create a secret in the vault server pod using KV secrets engine . This secret will be injected into the application pod’s volume which is not aware of the vault.

kubectl exec vault-0 -- vault kv put secret/helloworld username=user1 password=pass

• Retrieve the secret for verification from the vault server pod

kubectl exec vault-0 -- vault kv get secret/helloworld

Deploy Application and Access Vault Secrets

• This is a web-service along with service account which we attach to the policy earlier. This will help in specifying the secret an application is allowed to access.

# app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: app
      containers:
      - name: app
        image: jweissig/app:0.0.1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app
  labels:
    app: vault-agent-demo

kubectl apply -f app.yaml

kubectl get deploy,pods

• Do exec into the above deployment pod and verify that there are not secrets mounted at /vault/secrets location

kubectl exec -it deploy/app -c app -- ls -l /vault/secrets

This is because we haven’t applied annotations to the application pod which will enable the injection of secrets into the pod.

• Patch the annotations to the existing application pod

# patch-basic-annotations.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secret/helloworld"
        vault.hashicorp.com/role: "myapp"

These annotations define a qualified structure of the deployment schema and are prefixed with vault.hashicorp.com.

• agent-inject enables the Vault Agent Injector service
• role is the Vault Kubernetes authentication role
• agent-inject-secret-FILEPATH prefixes the path of secret file present in vault to be injected. 

kubectl patch deployment app --patch "$(cat patch-basic-annotations.yaml)"

• Now once again do exec into the application pod and check the mounted secrets at the location /vault/secrets

kubectl exec -it deploy/app -c app -- ls -l /vault/secrets

• Now, check the vault secret we injected through annotations

kubectl exec -it deploy/app -c app -- cat /vault/secrets/helloworld

As we are able to access the secrets inside the pod. This happened here because on applying the patch, Kubernetes webhook intercepted and altered the pod definition which helps in including init-container to insert secrets and vault agent sidecar to keep the data in sync through the application lifecycle.

• As secrets are not quite readable inside the pod so to retrieve them in a proper format, we can specify the format in the patch annotations.

# patch-template-annotations.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secret/helloworld"
        vault.hashicorp.com/agent-inject-template-helloworld: |
          {{- with secret "secret/helloworld" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end }}
        vault.hashicorp.com/role: "myapp"

kubectl patch deployment app --patch "$(cat patch-template-annotations.yaml)"

• Now, retrieve the secrets 

kubectl exec -it deploy/app -c app -- cat /vault/secrets/helloworld

Conclusion

In this hands-on lab, we have learned about sidecar injection through vault in Kubernetes.

What next?

Next, we will see how to generate dynamic secrets in the vault.