A practical walkthrough in exploring namespaces in relation to docker and kubernetes.

As you probably know, containers running on the same host, share the Linux kernel. That’s why a container image does not contain a kernel, only software and tools that make up a distro like for example a package manager.

So, if containers share the same kernel, how are they ’isolated’? Technically, containers are processes that can use limited resources (controlled by cgroups) and isolated by Linux namespaces.

Namespaces are a feature of the kernel that partitions kernel resources such that one set of processes sees one set of resources while another set of processes sees a different set of resources.

Examples of such resources are process IDs, hostnames, user IDs, file names, and some names associated with network access, and interprocesses communication. (cfr. https://en.wikipedia.org/wiki/Linux_namespaces)

So, let’s give it a try.

Exploring namespaces on a docker host

Let’s get started!

Install docker

apt update; apt install -y docker.io

docker version

Create a container and find the pid

Let’s spin-up a simple nginx container.

docker run -d --name demo-nginx nginx

We can find the process ID using different ways

PID=$(docker inspect -f '{{.State.Pid}}' demo-nginx); echo $PID

or in a more verbose way

sudo ps aux | grep -i nginx

Finding the linux namespaces of the container

There are several ways to find the namespace id’s, but the most effective way I found is:

sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | grep -i $PID

Now we found the namespace id, let’s focus on the netns id and store the id into an env variable for further reference.

NETNS=$(sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | grep -i -m 1 $PID | awk '{print $2}'); \
echo $NETNS

Now, let’s find all processes that share the netns.

sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | \
grep $NETNS

We now found all the processes that share the namespace. These are all the processes currently running inside the context of the container. You can easily add another process to an existing namespace using the docker cli. For example:

docker run -d --net=container:demo-nginx busybox /bin/sh -c "wget http://127.0.0.1; sleep 120"

This will execute the curl command and add the sleep process while sharing the existing netns namespace.

sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | \
grep $NETNS

We are sharing the netns namespace in these examples. This means that all containers share the networking stack. This is the reason why the containers can communicate over 127.0.0.1. To verify the curl command was successful.

docker logs demo-nginx

Let’s cleanup

docker stop demo-nginx; docker rm demo-nginx

Exploring namespaces on a kubernetes node

Create a k8s pod and find the pid

 kubectl run --image nginx www-demo

This pod is running on a single node cluster. (Note: in a real-life scenario, you must connect to the node the pod is actually running on). You can identify the node easily.

kubectl get po -o wide

On the node the pod is running, we can find the process id.

ps -ax -n -o pid,netns,cmd  | grep -i -m 1 "master process nginx" 

Finding the linux namespaces of the pod

Let’s store the netns in an env variable.

NETNS=$(ps -ax -n -o pid,netns,cmd | grep -i -m 1  "master process nginx" | awk '{print $2}'); echo $NETNS

And find all the processes that share the netns namespace

sudo ps -ax -n -o pid,netns,utsns,ipcns,mntns,pidns,cmd | \
grep $NETNS

We now found all the processes in the context of the pod, but what we see is even more interesting. We found the /pause container. It’s a container which holds the network netns, utsnsand ipcns namespace for the pod. Kubernetes creates the ‘pause’ containers to acquire for example the respective pod’s IP address and share it with all other containers that join that pod.

Hacking the pod via namespaces

Could we modify the content of the content of www-demo ? Let’s verify the pod is still there.

kubectl get po 

Let’s quickly store the PID.

PID=$(ps -ax -n -o pid,netns,cmd | grep -i -m 1  "master process nginx" | awk '{print $1}'); echo $PID

nsenter will help us navigate into the namespaces of the pod

PID=$(ps -ax -n -o pid,netns,cmd | grep -i -m 1  "master process nginx" | awk '{print $1}'); echo $PID
sudo nsenter --target $PID --mount --uts --ipc --net --pid

We now have full access to the pod, including the filesystem. Let’s try to add a file to the nginx container.

echo  "Hacking" >/usr/share/nginx/html/hacking.html; exit

Let exec into the pod and verify !

kubectl exec -it www-demo -- curl 127.0.0.1/hacking.html

Conclusion

I hope this small tutorial helps in better understanding how containers and pods relate to Linux namespaces and sheds a light on how isolation works in containerised environments.