How to TCPdump in Kubernetes !
In this lab, we will focus on how we can leverage tcpdump in a Kubernetes environment.
The technique discussed in How to TCPdump effectively in Docker!
might be a possible solution, but there are subtle differences.
- Kubernetes is typically multi-node, so you have to figure out the correct node
- docker is not a common runtime for kubernetes anymore, so you need to look and deep-dive in containerd or cri-o
- Kubernetes uses pods, so you need to find the corresponding containers first
This complexity inspired me to investigate other options and have been described in different tutorials (see References)
In the next labs, we'll discuss two mechanisms to get started.
Deployment patching
Let's get started!
Create a deployment
Setup a small and simple test environment
kubectl create deploy www-demo --image nginx --replicas=2
Verify the pods are running
kubectl get pods
Quickly expose the deployment
kubectl expose deploy/www-demo --port 80
Take a quick look at a pod. We can see there is just a nginx
container inside the pod.
Our lab should be up and running by now. Let's proceed and update and patch our setup.
Create a patch file
The first technique is to inject a tcpdump
container into the pod, is by patching the deployment. Basicly, we'll update the deployment in etcd
and trigger a re-deploy.
So first create a patch.yaml
file.
cat <<EOF >patch.yaml
spec:
template:
spec:
containers:
- name: tcpdumper
image: docker.io/dockersec/tcpdump
EOF
Apply the patch
Next we need to apply the patch.yaml
file
kubectl patch deployment www-demo --patch "$(cat patch.yaml)"
Take some time to investigate, but please note:
- new pods are started, old pods are terminated
- pods have eventually 2/2 running containers
At this point, all our www-demo
pods are running a nginx
and tcpdump
container and are already collecting traffic !
Analyse the traffic
The following one-liner will attach to the first pod, but you can of course use pod/<pod-name>
instead of deploy/www-demo
, but it saves you from typing the long name ;-)
kubectl attach deploy/www-demo
In a Terminal2, let's generate some traffic. We are using siege
here, a well-know traffic generator.
kubectl run -it --rm --image dockersec/siege stress -- www-demo
Take a look in Terminal1 and off you go (press ctrl-c to stop)
You can see all the traffic that flows in/out of the pod. Note (since we did not specify any tcpdump parameters) you see names instead of addresses. They refence the pod or service name.
Let's roll back our changes and move on to the next technique.
kubectl rollout undo deploy/www-demo
Close Terminal2 and checkout the next part !
Ephemeral container
Verify if everything is still up and running.
kubectl get pods,svc
Just to make things easy, let's put a podname in a $POD
POD=$(kubectl get pod -o name | grep www-demo -m 1); echo $POD
The next command kubectl debug
enables the use of an ephemeral container
in kubernetes.
This feature is only available in very recent kubernetes versions (v1.23 as Beta).
Explaining the concept of an ephemeral containers would be way too advanced for the purpose of this labs, but a good starting point is https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/.
In short, an ephemeral container is special kind of container that can run with less
garantuees then normal containers (and less features) and can be added to a running pod without the need to restart the pod. So this ideal for debugging.
kubectl debug -it $POD --image=dockersec/tcpdump --target nginx -- sh
$POD
is the pod we are targetting--image
specifies the the image of our debug tools--target
specifies the container name inside the pod we want to attach to
If everything works out, you get a shell inside the debug container.
We can now run our debug tools.
tcpdump -n port 80
In a Terminal2, let's generate some traffic.
kubectl run -it --rm --image dockersec/siege stress -- www-demo
In Terminal1 we should observe a traffic capture !!
Conclusion
I hope this lab will contribute to better understanding kubernetes networking and troubleshooting network related issues. Two different techniques were covered to specifically capture traffic inside a pod. Feel free to reach out @xxradar to discuss.
References
Tools used
- siege - https://github.com/JoeDog/siege