How to TCPdump in Kubernetes !

How to TCPdump in Kubernetes !

In this lab, we will focus on how we can leverage tcpdump in a Kubernetes environment.
The technique discussed in How to TCPdump effectively in Docker!
might be a possible solution, but there are subtle differences.

• Kubernetes is typically multi-node, so you have to figure out the correct node
• docker is not a common runtime for kubernetes anymore, so you need to look and deep-dive in containerd or cri-o
• Kubernetes uses pods, so you need to find the corresponding containers first
  

This complexity inspired me to investigate other options and have been described in different tutorials (see References)

In the next labs, we'll discuss two mechanisms to get started.

• deployment patching
• ephemeral container
  

Deployment patching

Let's get started!

Create a deployment

Setup a small and simple test environment

kubectl create deploy www-demo --image nginx --replicas=2

Verify the pods are running

kubectl get pods 

Quickly expose the deployment

kubectl expose deploy/www-demo --port 80

Take a quick look at a pod. We can see there is just a nginx container inside the pod.

Our lab should be up and running by now. Let's proceed and update and patch our setup.

Create a patch file

The first technique is to inject a tcpdump container into the pod, is by patching the deployment. Basicly, we'll update the deployment in etcd and trigger a re-deploy.

So first create a patch.yaml file.

cat <<EOF >patch.yaml
spec:
  template:
    spec:
      containers:
      - name: tcpdumper
        image: docker.io/dockersec/tcpdump
EOF

Apply the patch

Next we need to apply the patch.yaml file

kubectl patch deployment  www-demo --patch "$(cat patch.yaml)"

Take some time to investigate, but please note:

• new pods are started, old pods are terminated
• pods have eventually 2/2 running containers
  

At this point, all our www-demo pods are running a nginx and tcpdump container and are already collecting traffic !

Analyse the traffic

The following one-liner will attach to the first pod, but you can of course use pod/<pod-name> instead of deploy/www-demo, but it saves you from typing the long name ;-)

kubectl attach deploy/www-demo 

In  a Terminal2, let's generate some traffic. We are using siege here, a well-know traffic generator.

kubectl run -it --rm --image dockersec/siege stress --  www-demo

Take a look in Terminal1 and off you go (press ctrl-c to stop)

You can see all the traffic that flows in/out of the pod. Note (since we did not specify any tcpdump parameters) you see names instead of addresses. They refence the pod or service name.

Let's roll back our changes and move on to the next technique.

kubectl rollout undo deploy/www-demo

Close Terminal2 and checkout the next part !

Ephemeral container

Verify if everything is still up and running.

kubectl get pods,svc

Just to make things easy, let's put a podname in a $POD

POD=$(kubectl get pod -o name | grep www-demo -m 1); echo $POD

The next command kubectl debug enables the use of an ephemeral container in kubernetes.
This feature is only available in very recent kubernetes versions (v1.23 as Beta).
Explaining the concept of an ephemeral containers would be way too advanced for the purpose of this labs, but a good starting point is https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/.

In short, an ephemeral container is special kind of container that can run with less
garantuees then normal containers (and less features) and can be added to a running  pod without the need to restart the pod. So this ideal for debugging.

kubectl debug -it $POD --image=dockersec/tcpdump --target nginx -- sh

• $POD is the pod we are targetting
• --image specifies the the image of our debug tools
• --target specifies the container name inside the pod we want to attach to

If everything works out, you get a shell inside the debug container.

We can now run our debug tools.

tcpdump -n port 80

In  a Terminal2, let's generate some traffic.

kubectl run -it --rm --image dockersec/siege stress --  www-demo

In Terminal1 we should observe a traffic capture !!

Conclusion

I hope this lab will contribute to better understanding kubernetes networking and troubleshooting network related issues. Two different techniques were covered to specifically capture traffic inside a pod. Feel free to reach out @xxradar to discuss.

References

• https://xxradar.medium.com/how-to-tcpdump-effectively-in-kubernetes-part-1-a1546b683d2f

• https://xxradar.medium.com/how-to-tcpdump-effectively-in-kubernetes-part-2-7e4127b42dc7

• https://xxradar.medium.com/tcpdump-nc-and-k8s-fun-1276414907b5

• https://xxradar.medium.com/how-to-tcpdump-using-ephemeral-containers-in-kubernetes-d066e6855785

Tools used

• siege - https://github.com/JoeDog/siege