In this lab, we will focus on how we can leverage tcpdump in a Kubernetes environment.
The technique discussed in How to TCPdump effectively in Docker!
might be a possible solution, but there are subtle differences.
- Kubernetes is typically multi-node, so you have to figure out the correct node
- docker is not a common runtime for kubernetes anymore, so you need to look and deep-dive in containerd or cri-o
- Kubernetes uses pods, so you need to find the corresponding containers first
This complexity inspired me to investigate other options and have been described in different tutorials (see References)
In the next labs, we’ll discuss two mechanisms to get started.
Pre-requisites
Doing hands-on is important to follow and learn as it helps us to understand the concepts easily. It is recommended to do hands-on on your local machine.
- Ensure you have access to a Kubernetes cluster with admin privileges on your local machine; if not, create one using Minikube or Kind.
- Make sure you have installed kubectl on your machine.
Deployment patching
Let’s get started!
Create a deployment
Setup a small and simple test environment
kubectl create deploy www-demo --image nginx --replicas=2
Verify the pods are running
kubectl get pods
Quickly expose the deployment
kubectl expose deploy/www-demo --port 80
Take a quick look at a pod. We can see there is just a nginx container inside the pod.
root@master:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
www-demo-8644c6bd5-7plwt 1/1 Running 0 48s
www-demo-8644c6bd5-k92wv 1/1 Running 0 48s
root@master:~#
root@master:~# kubectl describe pod www-demo-8644c6bd5-7plwt
Name: www-demo-8644c6bd5-7plwt
...
Status: Running
IP: 10.244.219.71
IPs:
IP: 10.244.219.71
Controlled By: ReplicaSet/www-demo-8644c6bd5
Containers:
nginx:
Container ID: containerd://0ffbebecee243d683dd2969862bd312761e8e33eee3452e2fc49203db899920f
Image: nginx
Image ID: docker.io/library/nginx@sha256:2bcabc23b45489fb0885d69a06ba1d648aeda973fae7bb981bafbb884165e514
Port: <none>
Host Port: <none>
State: Running
Started: Tue, 21 Jun 2022 19:12:47 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ktf5l (ro)
Conditions:
Type Status
Initialized True
Ready True
...Our lab should be up and running by now. Let’s proceed and update and patch our setup.
Create a patch file
The first technique is to inject a tcpdump container into the pod, is by patching the deployment. Basicly, we’ll update the deployment in etcd and trigger a re-deploy.
So first create a patch.yaml file.
cat <<EOF >patch.yaml
spec:
template:
spec:
containers:
- name: tcpdumper
image: docker.io/dockersec/tcpdump
EOF
Apply the patch
Next we need to apply the patch.yaml file
kubectl patch deployment www-demo --patch "$(cat patch.yaml)"
Take some time to investigate, but please note:
- new pods are started, old pods are terminated
- pods have eventually 2/2 running containers
root@master:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
www-demo-5df5fc48b6-2fw7w 0/2 ContainerCreating 0 10s
www-demo-5df5fc48b6-s9ds2 0/2 ContainerCreating 0 10s
www-demo-8644c6bd5-7plwt 1/1 Running 0 30m
www-demo-8644c6bd5-k92wv 1/1 Running 0 30m
root@master:~# kubectl describe pod www-demo-5df5fc48b6-2fw7w
error: the server doesn't have a resource type "www-demo-5df5fc48b6-2fw7w"
root@master:~# kubectl describe po www-demo-5df5fc48b6-2fw7w
Name: www-demo-5df5fc48b6-2fw7w
...
Status: Running
IP: 10.244.219.76
IPs:
IP: 10.244.219.76
Controlled By: ReplicaSet/www-demo-5df5fc48b6
Containers:
tcpdumper:
Container ID: containerd://2e3e42737f3c23bbc0aea5f0ebb9c1bcaf10a5271e7828996655723d184f29d6
Image: docker.io/dockersec/tcpdump
Image ID: docker.io/dockersec/tcpdump@sha256:ae1bbef80698f72f07017e5fe1742b1b758cbb833bddee648fc8889fa8f99fec
Port: <none>
Host Port: <none>
State: Running
Started: Tue, 21 Jun 2022 19:43:28 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4ntfd (ro)
nginx:
Container ID: containerd://a4f0cc77380af956602bf2ee2bda524084d50f1ef35c066c0a9f6ff7f85919a0
Image: nginx
Image ID: docker.io/library/nginx@sha256:2bcabc23b45489fb0885d69a06ba1d648aeda973fae7bb981bafbb884165e514
Port: <none>
Host Port: <none>
State: Running
Started: Tue, 21 Jun 2022 19:43:35 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4ntfd (ro)
Conditions:
Type Status
Initialized True
...
root@master:~#
At this point, all our www-demo pods are running a nginx and tcpdump container and are already collecting traffic !
Analyse the traffic
The following one-liner will attach to the first pod, but you can of course use pod/ instead of deploy/www-demo, but it saves you from typing the long name 😉
kubectl attach deploy/www-demo
In a Terminal2, let’s generate some traffic. We are using siege here, a well-know traffic generator.
kubectl run -it --rm --image dockersec/siege stress -- www-demo
Take a look in Terminal1 and off you go (press ctrl-c to stop)
root@master:~# kubectl attach deploy/www-demo Defaulted container "tcpdumper" out of: tcpdumper, nginx If you don't see a command prompt, try pressing enter. 19:43:40.958497 IP6 fe80::64ee:61ff:fe59:151 > ff02::2: ICMP6, router solicitation, length 16 19:43:41.663302 ARP, Request who-has 169.254.1.1 tell www-demo-5df5fc48b6-s9ds2, length 28 19:43:41.663321 ARP, Reply 169.254.1.1 is-at ee:ee:ee:ee:ee:ee (oui Unknown), length 28 19:43:41.667234 IP kube-dns.kube-system.svc.cluster.local.53 > www-demo-5df5fc48b6-s9ds2.49233: 21415 NXDomain 0/1/0 (166) 19:43:42.686865 IP www-demo-5df5fc48b6-s9ds2.45563 > kube-dns.kube-system.svc.cluster.local.53: 44535+ PTR? 1.1.254.169.in-addr.arpa. (42) 19:43:42.691047 IP kube-dns.kube-system.svc.cluster.local.53 > www-demo-5df5fc48b6-s9ds2.45563: 44535 NXDomain 0/0/0 (42) 19:43:42.691696 IP www-demo-5df5fc48b6-s9ds2.38137 > kube-dns.kube-system.svc.cluster.local.53: 63521+ PTR? 10.0.96.10.in-addr.arpa. (41) 19:43:42.692170 IP kube-dns.kube-system.svc.cluster.local.53 > www-demo-5df5fc48b6-s9ds2.38137: 63521*- 1/0/0 PTR kube-dns.kube-system.svc.cluster.local. (116) 19:43:46.850428 ARP, Request who-has www-demo-5df5fc48b6-s9ds2 tell 10-0-0-97.kubernetes.default.svc.cluster.local, length 28 19:43:46.850574 ARP, Reply www-demo-5df5fc48b6-s9ds2 is-at 66:ee:61:59:01:51 (oui Unknown), length 28 19:43:47.806812 IP www-demo-5df5fc48b6-s9ds2.47939 > kube-dns.kube-system.svc.cluster.local.53: 54285+ PTR? 97.0.0.10.in-addr.arpa. (40) 19:44:13.726510 IP6 fe80::64ee:61ff:fe59:151 > ff02::2: ICMP6, router solicitation, length 16 19:45:17.214584 IP6 fe80::64ee:61ff:fe59:151 > ff02::2: ICMP6, router solicitation, length 16 19:47:15.998600 IP6 fe80::64ee:61ff:fe59:151 > ff02::2: ICMP6, router solicitation, length 16 19:51:05.374555 IP6 fe80::64ee:61ff:fe59:151 > ff02::2: ICMP6, router solicitation, length 16 19:58:18.250106 IP 10.244.219.82.43774 > www-demo-5df5fc48b6-s9ds2.80: Flags [S], seq 4235057989, win 64800, options [mss 1440,sackOK,TS val 211629904 ecr 0,nop,wscale 7], length 0 19:58:18.250120 IP 10.244.219.82.43774 > www-demo-5df5fc48b6-s9ds2.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 211629904 ecr 544364641], length 0 19:58:18.250242 IP 10.244.219.82.43776 > www-demo-5df5fc48b6-s9ds2.80: Flags [S], seq 2411996267,
You can see all the traffic that flows in/out of the pod. Note (since we did not specify any tcpdump parameters) you see names instead of addresses. They refence the pod or service name.
Let’s roll back our changes and move on to the next technique.
kubectl rollout undo deploy/www-demo
Close Terminal2 and checkout the next part !
Ephemeral container
Verify if everything is still up and running.
kubectl get pods,svc
Just to make things easy, let’s put a podname in a $POD
POD=$(kubectl get pod -o name | grep www-demo -m 1); echo $POD
The next command kubectl debug enables the use of an ephemeral container in kubernetes.
This feature is only available in very recent kubernetes versions (v1.23 as Beta).
Explaining the concept of an ephemeral containers would be way too advanced for the purpose of this labs, but a good starting point is https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/.
In short, an ephemeral container is special kind of container that can run with less
garantuees then normal containers (and less features) and can be added to a running pod without the need to restart the pod. So this ideal for debugging.
kubectl debug -it $POD --image=dockersec/tcpdump --target nginx -- sh
- $POD is the pod we are targetting
- –image specifies the the image of our debug tools
- –target specifies the container name inside the pod we want to attach to
If everything works out, you get a shell inside the debug container.
We can now run our debug tools.
tcpdump -n port 80
In a Terminal2, let’s generate some traffic.
kubectl run -it --rm --image dockersec/siege stress -- www-demo
In Terminal1 we should observe a traffic capture !!
09:28:02.353225 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [S], seq 1942359967, win 64800, options [mss 1440,sackOK,TS val 816593060 ecr 0,nop,wscale 7], length 0 09:28:02.353251 IP www-demo-8644c6bd5-4q2g2.80 > 10.244.219.87.60534: Flags [S.], seq 2976241644, ack 1942359968, win 64260, options [mss 1440,sackOK,TS val 1284658... 09:28:02.353302 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 816593060 ecr 1284658392], length 0 09:28:02.353410 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [P.], seq 1:149, ack 1, win 507, options [nop,nop,TS val 816593060 ecr 1284658392], length 148: HTTP: GET / HTTP/1.1 09:28:02.353420 IP www-demo-8644c6bd5-4q2g2.80 > 10.244.219.87.60534: Flags [.], ack 149, win 501, options [nop,nop,TS val 1284658392 ecr 816593060], length 0 09:28:02.353564 IP www-demo-8644c6bd5-4q2g2.80 > 10.244.219.87.60534: Flags [P.], seq 1:234, ack 149, win 501, options [nop,nop,TS val 1284658392 ecr 816593060], length 233: HTTP: HTTP/1.1 200 OK 09:28:02.353640 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [.], ack 234, win 506, options [nop,nop,TS val 816593060 ecr 1284658392], length 0 09:28:02.353674 IP www-demo-8644c6bd5-4q2g2.80 > 10.244.219.87.60534: Flags [P.], seq 234:849, ack 149, win 501, options [nop,nop,TS val 1284658392 ecr 816593060], length 615: HTTP 09:28:02.353706 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [.], ack 849, win 502, options [nop,nop,TS val 816593060 ecr 1284658392], length 0 09:28:02.354285 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [F.], seq 149, ack 849, win 502, options [nop,nop,TS val 816593061 ecr 1284658392], length 0 09:28:02.354300 IP 10.244.219.87.60536 > www-demo-8644c6bd5-4q2g2.80: Flags [
Conclusion
I hope this lab will contribute to better understanding kubernetes networking and troubleshooting network related issues. Two different techniques were covered to specifically capture traffic inside a pod. Feel free to reach out @xxradar to discuss.
References
- https://xxradar.medium.com/how-to-tcpdump-effectively-in-kubernetes-part-1-a1546b683d2f
- https://xxradar.medium.com/how-to-tcpdump-effectively-in-kubernetes-part-2-7e4127b42dc7
- https://xxradar.medium.com/tcpdump-nc-and-k8s-fun-1276414907b5
- https://xxradar.medium.com/how-to-tcpdump-using-ephemeral-containers-in-kubernetes-d066e6855785
Tools used
- siege – https://github.com/JoeDog/siege
