How to TCPdump in Kubernetes !

An introduction and references for tcpdump troubleshooting in kubernetes.

In this lab, we will focus on how we can leverage tcpdump in a Kubernetes environment.
The technique discussed in How to TCPdump effectively in Docker!
might be a possible solution, but there are subtle differences.

  • Kubernetes is typically multi-node, so you have to figure out the correct node
  • docker is not a common runtime for kubernetes anymore, so you need to look and deep-dive in containerd or cri-o
  • Kubernetes uses pods, so you need to find the corresponding containers first

This complexity inspired me to investigate other options and have been described in different tutorials (see References)

In the next labs, we’ll discuss two mechanisms to get started.

Pre-requisites

Doing hands-on is important to follow and learn as it helps us to understand the concepts easily. It is recommended to do hands-on on your local machine.

  • Ensure you have access to a Kubernetes cluster with admin privileges on your local machine; if not, create one using Minikube or Kind.
  • Make sure you have installed kubectl on your machine.

Deployment patching

Let’s get started!

Create a deployment

Setup a small and simple test environment

kubectl create deploy www-demo --image nginx --replicas=2

Verify the pods are running

kubectl get pods

Quickly expose the deployment

kubectl expose deploy/www-demo --port 80

Take a quick look at a pod. We can see there is just a nginx container inside the pod.

root@master:~# kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
www-demo-8644c6bd5-7plwt   1/1     Running   0          48s
www-demo-8644c6bd5-k92wv   1/1     Running   0          48s
root@master:~#

root@master:~# kubectl describe pod www-demo-8644c6bd5-7plwt
Name:         www-demo-8644c6bd5-7plwt
...
Status:       Running
IP:           10.244.219.71
IPs:
  IP:           10.244.219.71
Controlled By:  ReplicaSet/www-demo-8644c6bd5
Containers:
  nginx:
    Container ID:   containerd://0ffbebecee243d683dd2969862bd312761e8e33eee3452e2fc49203db899920f
    Image:          nginx
    Image ID:       docker.io/library/nginx@sha256:2bcabc23b45489fb0885d69a06ba1d648aeda973fae7bb981bafbb884165e514
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 21 Jun 2022 19:12:47 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ktf5l (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
...

Our lab should be up and running by now. Let’s proceed and update and patch our setup.

Create a patch file

The first technique is to inject a tcpdump container into the pod, is by patching the deployment. Basicly, we’ll update the deployment in etcd and trigger a re-deploy.

So first create a patch.yaml file.

cat <<EOF >patch.yaml
spec:
  template:
    spec:
      containers:
      - name: tcpdumper
        image: docker.io/dockersec/tcpdump
EOF

Apply the patch

Next we need to apply the patch.yaml file

kubectl patch deployment  www-demo --patch "$(cat patch.yaml)"

Take some time to investigate, but please note:

  • new pods are started, old pods are terminated
  • pods have eventually 2/2 running containers
root@master:~# kubectl get pods
NAME                        READY   STATUS              RESTARTS   AGE
www-demo-5df5fc48b6-2fw7w   0/2     ContainerCreating   0          10s
www-demo-5df5fc48b6-s9ds2   0/2     ContainerCreating   0          10s
www-demo-8644c6bd5-7plwt    1/1     Running             0          30m
www-demo-8644c6bd5-k92wv    1/1     Running             0          30m

root@master:~# kubectl describe pod www-demo-5df5fc48b6-2fw7w
error: the server doesn't have a resource type "www-demo-5df5fc48b6-2fw7w"
root@master:~# kubectl describe po www-demo-5df5fc48b6-2fw7w
Name:         www-demo-5df5fc48b6-2fw7w
...
Status:       Running
IP:           10.244.219.76
IPs:
  IP:           10.244.219.76
Controlled By:  ReplicaSet/www-demo-5df5fc48b6
Containers:
  tcpdumper:
    Container ID:   containerd://2e3e42737f3c23bbc0aea5f0ebb9c1bcaf10a5271e7828996655723d184f29d6
    Image:          docker.io/dockersec/tcpdump
    Image ID:       docker.io/dockersec/tcpdump@sha256:ae1bbef80698f72f07017e5fe1742b1b758cbb833bddee648fc8889fa8f99fec
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 21 Jun 2022 19:43:28 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4ntfd (ro)
  nginx:
    Container ID:   containerd://a4f0cc77380af956602bf2ee2bda524084d50f1ef35c066c0a9f6ff7f85919a0
    Image:          nginx
    Image ID:       docker.io/library/nginx@sha256:2bcabc23b45489fb0885d69a06ba1d648aeda973fae7bb981bafbb884165e514
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 21 Jun 2022 19:43:35 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4ntfd (ro)
Conditions:
  Type              Status
  Initialized       True
 ...
root@master:~#

At this point, all our www-demo pods are running a nginx and tcpdump container and are already collecting traffic !

Analyse the traffic

The following one-liner will attach to the first pod, but you can of course use pod/ instead of deploy/www-demo, but it saves you from typing the long name 😉

kubectl attach deploy/www-demo

In a Terminal2, let’s generate some traffic. We are using siege here, a well-know traffic generator.

kubectl run -it --rm --image dockersec/siege stress --  www-demo

Take a look in Terminal1 and off you go (press ctrl-c to stop)

root@master:~# kubectl attach deploy/www-demo
Defaulted container "tcpdumper" out of: tcpdumper, nginx
If you don't see a command prompt, try pressing enter.
19:43:40.958497 IP6 fe80::64ee:61ff:fe59:151 > ff02::2: ICMP6, router solicitation, length 16
19:43:41.663302 ARP, Request who-has 169.254.1.1 tell www-demo-5df5fc48b6-s9ds2, length 28
19:43:41.663321 ARP, Reply 169.254.1.1 is-at ee:ee:ee:ee:ee:ee (oui Unknown), length 28
19:43:41.667234 IP kube-dns.kube-system.svc.cluster.local.53 > www-demo-5df5fc48b6-s9ds2.49233: 21415 NXDomain 0/1/0 (166)
19:43:42.686865 IP www-demo-5df5fc48b6-s9ds2.45563 > kube-dns.kube-system.svc.cluster.local.53: 44535+ PTR? 1.1.254.169.in-addr.arpa. (42)
19:43:42.691047 IP kube-dns.kube-system.svc.cluster.local.53 > www-demo-5df5fc48b6-s9ds2.45563: 44535 NXDomain 0/0/0 (42)
19:43:42.691696 IP www-demo-5df5fc48b6-s9ds2.38137 > kube-dns.kube-system.svc.cluster.local.53: 63521+ PTR? 10.0.96.10.in-addr.arpa. (41)
19:43:42.692170 IP kube-dns.kube-system.svc.cluster.local.53 > www-demo-5df5fc48b6-s9ds2.38137: 63521*- 1/0/0 PTR kube-dns.kube-system.svc.cluster.local. (116)
19:43:46.850428 ARP, Request who-has www-demo-5df5fc48b6-s9ds2 tell 10-0-0-97.kubernetes.default.svc.cluster.local, length 28
19:43:46.850574 ARP, Reply www-demo-5df5fc48b6-s9ds2 is-at 66:ee:61:59:01:51 (oui Unknown), length 28
19:43:47.806812 IP www-demo-5df5fc48b6-s9ds2.47939 > kube-dns.kube-system.svc.cluster.local.53: 54285+ PTR? 97.0.0.10.in-addr.arpa. (40)
19:44:13.726510 IP6 fe80::64ee:61ff:fe59:151 > ff02::2: ICMP6, router solicitation, length 16
19:45:17.214584 IP6 fe80::64ee:61ff:fe59:151 > ff02::2: ICMP6, router solicitation, length 16
19:47:15.998600 IP6 fe80::64ee:61ff:fe59:151 > ff02::2: ICMP6, router solicitation, length 16
19:51:05.374555 IP6 fe80::64ee:61ff:fe59:151 > ff02::2: ICMP6, router solicitation, length 16
19:58:18.250106 IP 10.244.219.82.43774 > www-demo-5df5fc48b6-s9ds2.80: Flags [S], seq 4235057989, win 64800, options [mss 1440,sackOK,TS val 211629904 ecr 0,nop,wscale 7], length 0
19:58:18.250120 IP 10.244.219.82.43774 > www-demo-5df5fc48b6-s9ds2.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 211629904 ecr 544364641], length 0
19:58:18.250242 IP 10.244.219.82.43776 > www-demo-5df5fc48b6-s9ds2.80: Flags [S], seq 2411996267,

You can see all the traffic that flows in/out of the pod. Note (since we did not specify any tcpdump parameters) you see names instead of addresses. They refence the pod or service name.

Let’s roll back our changes and move on to the next technique.

kubectl rollout undo deploy/www-demo

Close Terminal2 and checkout the next part !

Ephemeral container

Verify if everything is still up and running.

kubectl get pods,svc

Just to make things easy, let’s put a podname in a $POD

POD=$(kubectl get pod -o name | grep www-demo -m 1); echo $POD

The next command kubectl debug enables the use of an ephemeral container in kubernetes.
This feature is only available in very recent kubernetes versions (v1.23 as Beta).
Explaining the concept of an ephemeral containers would be way too advanced for the purpose of this labs, but a good starting point is https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/.

In short, an ephemeral container is special kind of container that can run with less
garantuees then normal containers (and less features) and can be added to a running pod without the need to restart the pod. So this ideal for debugging.

kubectl debug -it $POD --image=dockersec/tcpdump --target nginx -- sh
  • $POD is the pod we are targetting
  • –image specifies the the image of our debug tools
  • –target specifies the container name inside the pod we want to attach to

If everything works out, you get a shell inside the debug container.

We can now run our debug tools.

tcpdump -n port 80

In  a Terminal2, let’s generate some traffic.

kubectl run -it --rm --image dockersec/siege stress --  www-demo

In Terminal1 we should observe a traffic capture !!

09:28:02.353225 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [S], seq 1942359967, win 64800, options [mss 1440,sackOK,TS val 816593060 ecr 0,nop,wscale 7], length 0
09:28:02.353251 IP www-demo-8644c6bd5-4q2g2.80 > 10.244.219.87.60534: Flags [S.], seq 2976241644, ack 1942359968, win 64260, options [mss 1440,sackOK,TS val 1284658...
09:28:02.353302 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 816593060 ecr 1284658392], length 0
09:28:02.353410 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [P.], seq 1:149, ack 1, win 507, options [nop,nop,TS val 816593060 ecr 1284658392], length 148: HTTP: GET / HTTP/1.1
09:28:02.353420 IP www-demo-8644c6bd5-4q2g2.80 > 10.244.219.87.60534: Flags [.], ack 149, win 501, options [nop,nop,TS val 1284658392 ecr 816593060], length 0
09:28:02.353564 IP www-demo-8644c6bd5-4q2g2.80 > 10.244.219.87.60534: Flags [P.], seq 1:234, ack 149, win 501, options [nop,nop,TS val 1284658392 ecr 816593060], length 233: HTTP: HTTP/1.1 200 OK
09:28:02.353640 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [.], ack 234, win 506, options [nop,nop,TS val 816593060 ecr 1284658392], length 0
09:28:02.353674 IP www-demo-8644c6bd5-4q2g2.80 > 10.244.219.87.60534: Flags [P.], seq 234:849, ack 149, win 501, options [nop,nop,TS val 1284658392 ecr 816593060], length 615: HTTP
09:28:02.353706 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [.], ack 849, win 502, options [nop,nop,TS val 816593060 ecr 1284658392], length 0
09:28:02.354285 IP 10.244.219.87.60534 > www-demo-8644c6bd5-4q2g2.80: Flags [F.], seq 149, ack 849, win 502, options [nop,nop,TS val 816593061 ecr 1284658392], length 0
09:28:02.354300 IP 10.244.219.87.60536 > www-demo-8644c6bd5-4q2g2.80: Flags [

Conclusion

I hope this lab will contribute to better understanding kubernetes networking and troubleshooting network related issues. Two different techniques were covered to specifically capture traffic inside a pod. Feel free to reach out @xxradar to discuss.

References

Tools used

Join Our Newsletter

Share this article:

Philippe Bogaerts

Philippe Bogaerts

All Posts
Table of Contents

Want to Master your Cloud Career ?

Show me How

We are on a mission to empower careers and uplift businesses through modern cloud education, ensuring excellence in today’s digital landscape.

GitHub Rss

Company

Resources

Subscribe to Newsletter

© Copyright CloudYuga 2023. All rights reserved.

Terms & Conditions

Privacy Policy

Refund Policy