Introduction to Confidential Containers

Introduction to Confidential Containers

28 September 2022
cloud native
kubernetes
Cncf
CoCo
Confidential Containers

Get familiar with Kata containers based Confidential Containers stack

Confidential Containers (CoCo) is a CNCF sandbox project which aims to integrate existing Confidential Computing (CC) infrastructure support and technologies with the cloud-native world.

The following references are a good read:

In this lab, we’ll be deploying the Kata containers based CoCo stack as shown in the diagram below.

Further, we'll demonstrate the container image management capability of the CoCo stack, whereby the container images are downloaded inside the Kata VM and not on the Kubernetes cluster node.

Figure 1: Kata Containers based CoCo Stack
Figure 1: Kata Containers based CoCo Stack
Please note that in this lab we are deploying the CoCo stack on a regular (non CC) hardware to help you get familiar with the CoCo stack.

Install a KinD Kubernetes cluster

  • Install docker as a prerequisite for KinD Cluster setup
sudo apt update && sudo apt install docker.io -y
  • Install kubectl CLI to interact with the cluster
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update && sudo apt-get install -y kubectl
  • Install KinD from its binaries
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.16.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
  • Verify the KinD installation
kind version
  • Now, create a single node Kubernetes cluster via KinD and give its name coco
kind create cluster --name coco

Wait for some time to get it provisioned.

  • Check the cluster node's status and verify the Kubernetes version is greater than 1.24 and contained is CRI.
kubectl get nodes -o wide 

Setup the CoCo Stack

Setting up the CoCo stack is super easy, courtesy of the CoCo operator

  • The CoCo operator requires at least one cluster node having the label node-role.kubernetes.io/worker= . Set the environment variable NODENAME with the cluster node name.
export NODENAME=coco-control-plane
kubectl label node $NODENAME node-role.kubernetes.io/worker=

Verify the role on the node through

kubectl get nodes -o wide 
  • Deploy the CoCo Operator by running the following command:
kubectl apply -f https://raw.githubusercontent.com/confidential-containers/operator/main/deploy/deploy.yaml

The operator deploys all resources under the confidential-containers-system namespace.

Wait until each pod has the STATUS of Running.

kubectl get pods -n confidential-containers-system --watch

The operator is responsible for creating the custom resource definition (CRD), which is then used for creating a custom resource (CR).

The operator creates the ccruntime CRD as can be observed by running the following command:

kubectl get crd | grep ccruntime

The complete CRD can be seen by running the following command:

kubectl explain --recursive=true ccruntimes.confidentialcontainers.org
You can also see the details of the CRD in the following source file: https://github.com/confidential-containers/operator/blob/main/api/v1beta1/ccruntime_types.go#L90 .
  • Create the CR by running the following command:
kubectl apply -f https://raw.githubusercontent.com/confidential-containers/operator/main/config/samples/ccruntime.yaml

This triggers the deployment of the Kata containers based CoCo stack on the KinD cluster.

Wait until each pod has the STATUS of Running.

kubectl get pods -n confidential-containers-system --watch
  • Verify that the RuntimeClasses got created.
kubectl get runtimeclass

Following is a brief explanation of each of the RuntimeClasses:

kata - standard kata runtime using the QEMU hypervisor including all CoCo building blocks for a non Confidential Computing (CC) hardware

kata-clh - standard kata runtime using the cloud hypervisor including all CoCo building blocks for a non CC hardware

kata-clh-tdx - using the Cloud Hypervisor, with TD-Shim, and support for Intel TDX CC hardware

kata-qemu - same as kata

kata-qemu-tdx - using QEMU, with TDVF, and support for Intel TDX CC hardware

kata-qemu-sev - using QEMU, and support for AMD SEV hardware

Since we are running the Kata containers based CoCo stack on a non-CC hardware, we can use only kata and kata-clh.RuntimeClasses. Further, only Cloud Hypervisor works with Kata on a KinD Kubernetes cluster. Consequently, we’ll be using only kata-clh RuntimeClass.

Creating a sample workload

The sample workload is intended to show how the CoCo building blocks work together.

A key aspect of CoCo when using Kata containers is that the container images get downloaded inside the Kata VM and not on the cluster node as the cluster node is not trusted. Kata VM is the trusted entity from an end-user standpoint.

Further, the container images can be signed and encrypted with signature verification and decryption inside the Kata VM. The decryption keys are made available to the Kata VM by an attestation service after successful attestation. You can read more about the attestation process in the following blog. In a future lab guide we'll cover the attestation process.

  • For the sample workload, we'll use the bitnami/nginx image described in the following nginx.yaml:
  • Before creating the pod from the above manifest, first, verify that the container image doesn’t exist on the cluster node through crictl .

Login to the KinD cluster node by executing the following command in the terminal.

CID=$(docker ps -qf "name=coco-control-plane")
docker exec -it $CID bash

You should be able to get a shell to the KinD cluster node (as shown below).

root@coco-control-plane:/# 

Now, verify whether the container image exists on the KinD cluster node by executing the following command in the terminal.

crictl image ls | grep bitnami/nginx

You should see an empty output.

  • Now create the POD by executing the following command:
kubectl apply -f nginx.yaml
  • Ensure the pod was created successfully (in running state):
kubectl get pods

Now re-verify that the container image doesn’t exist on the KinD cluster node by running the following command in the container shell as described previously:

crictl image ls | grep bitnami/nginx

Again, you should see an empty output.

If you have made it here, give yourself a shoutout :-). Do not worry if you are stuck with the instructions or need details on the CoCo stack. Help is right around the corner.

How likely are you going to recommend this lab to your friends or colleagues?

Unlikely
Likely

Leave a comment:

About the Authors

Pradipta Banerjee

Pradipta Banerjee

Senior Principal Software Engineer, Red Hat

Pradipta is currently working on container isolation and confidential computing. He is a strong believer in self-learning and hands-on problem-solving. Connect with him for any help with container security, digitization, or technology adoption for improving livelihoods

Oshi Gupta

Oshi Gupta

DevOps Engineer & Technical Writer, CloudYuga

Oshi Gupta is currently working as DevOps Engineer and Technical Writer at CloudYuga. She is working on Kubernetes and different cloud-native technologies.