Investigating eBPF-Driven Observability in Kubernetes Environments for Sensitive File Access

The container orchestration standard has been taken over by Kubernetes. However, as it doesn’t offer a default observability for security to study the attacks, it raises questions about how organisations should protect their production environment. This issue is resolved with eBPF.

Currently, one of the most popular subjects being addressed in the industry is eBPF. Not only developers that work with the Linux kernel but the cloud-native community is also attracted to it.

• Tetragon is an open-source project of Cilium that offers run-time security, deep observability, and kernel-level transparency using eBPF.
•  Parseable is a lightweight, high-performance log analytics stack for developers.

How we will achieve Tetragon logs in the Parseable UI will be the topic of discussion in this hands-on lab. Let’s first understand a bit about eBPF before moving further.

What is eBPF?

eBPF is a technology that enables the Linux kernel to be extended at runtime, i.e. without modifying the kernel source code or rebooting it. It has been noted that eBPF does to Linux exactly what Javascript does to HTML.

The extended Berkeley packet filter, or eBPF, allows us to dynamically program the Linux kernel and obtain insights. Generally, eBPF functions in networking, security, and observability.

You can refer to our hands-on lab eBPF- an Overview for more detail.

What is Tetragon?

Tetragon is a powerful Kubernetes-aware security observability and runtime enforcement tool that applies TracingPolicy and filtering directly with eBPF. It ensures the reduction of observation overhead by tracking of any process and real-time policy enforcement.

As shown in the above figure, Tetragon is deployed to the Kubernetes cluster. Tetragon uses hook points in eBPF programs to observe kernel calls. If any TracingPolicy is applied, it filters the logs as per the policy configuration. When any event is filtered using TracingPolicy, it stores Tetragon logs locally or uses a webhook backend like Parseable to collect logs. You can refer to our hands-on lab Tetragon: Empowering Kubernetes Security via Dynamic eBPF Insights for more detail. 

Implementation of Tetragon Logs in Parseable through Vector

We’ll implement Tetragon logs using TracingPolicy, the Tetragon server stores them in a local file system, and configure Vector, a lightweight, ultra-fast tool to send logs to the Parseable server. At last, we will verify Tetragon logs in Parseable UI with sensitive file access through the Kubernetes pod.

Here is the high-level architecture that shows what we would achieve from this hands-on lab.

Please click on the LAB-SETUP button to get the Kubernetes cluster ready within a few seconds. We will perform the following tasks to understand how Parseable helps us in auditing Tetragon logs.

• Enable Tetragon in the Kubernetes cluster which stores logs locally.
• Send Tetragon logs in Parseable through Vector.
• Sensitive files will be accessed from the pod then alerts will be generated through Parseable.

Tetragon Installation

We will install Tetragon using Helm. 

helm repo add cilium https://helm.cilium.io

helm repo update

helm install tetragon cilium/tetragon -n kube-system

Wait for a moment,

kubectl get ds tetragon -n kube-system

• To view the logs,

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f

Optional

• To get the events in better format, install tetra cli for this.

curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-linux-amd64.tar.gz | tar -xz

sudo mv tetra /usr/local/bin

• Observe the events with tetra command

kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact

Extract Tetragon Logs in Parseable via vector

Parseable installation

Parseable is a lightweight, high-speed, logging and observability platform for cloud-native applications. 

• To install the Parseable in our Kubernetes cluster, run the following script that is attached to the lab setup. 

bash parseable-installation.sh

• Wait for a few minutes for the Parseable pod to get started and verify the Parseable pods are running in a parseable namespace.

kubectl get pods -n parseable

Now, the application is ready and exposed to the relevant port (app-port-31000). We’ll now use Vector agent, configure Tetragon logs as its data source, and sink it to the Parseable server.

Follow the installation guide to install Parseable on Kubernetes cluster in detail.

Install and configure Vector Agent

We’ll install Vector through Helm. We have configured a vector-tetragon-values.yaml file (It is attached to the lab setup) such that the vector can access the tetragon.log file.

helm repo add vector https://helm.vector.dev

helm install vector vector/vector --namespace vector --create-namespace --values vector-tetragon-values.yaml

It will take a few minutes for the Vector pod to get started. Verify the vector pod is running in a vector namespace.

kubectl get pods -n vector

Now Vector is ready to send the events stored in /var/run/cilium/tetragon/tetragon.log file to the Parseable tetrademo log stream. Once this is done, you can verify the log events in Parseable UI (Click on app-port-31000).

Track Sensitive File Access with Parseable

Although integrating Tetragon logs with Parseable has several advantages, we will concentrate on one of the most notable use cases, namely auditing Tetragon events when accessing sensitive files like /etc/shadow.

Create a Pod to Access File from the Host

• Create a pod with privileged access Following pod configuration is in lab setup.

vim pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nsenter
spec:
  containers:
  - command:
    - /nsenter
    - --all
    - --target=1
    - --
    - su
    - '-'
    image: alexeiled/nsenter:2.34
    name: nsenter
    securityContext:
      privileged: true
    stdin: true
    tty: true
  hostNetwork: true
  hostPID: true

kubectl apply -f pod.yaml

• To list the pods,

kubectl get pods

The next step is to apply policy on Tetragon logs so it will extract particular types of logs.

Apply TracingPolicy

• Apply following TracingPolicy for read/write access. Please open the tracingpolicy.yaml file that is attached to the lab setup. 

vim tracingpolicy.yaml

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "file-monitoring"
spec:
  kprobes:
  - call: "security_file_permission"
    syscall: false
    return: true
    args:
    - index: 0
      type: "file" # (struct file *) used for getting the path
    - index: 1
      type: "int" # 0x04 is MAY_READ, 0x02 is MAY_WRITE
    returnArg:
      index: 0
      type: "int"
    returnArgAction: "Post"
    selectors:
    - matchArgs:      
      - index: 0
        operator: "Prefix"
        values:
        - "/etc/passwd"   # filenames to filter for
  - call: "security_mmap_file"
    syscall: false
  .........
        values:
        - "/etc/passwd"   # filenames to filter for
  - call: "security_path_truncate"
    syscall: false
   .........
        values:
        - "/etc/passwd"   # filenames to filter for

It will monitor a specific file /etc/passwd. Here, the kprobe is a hook point that hooks on the following kernel functions from where the file can be accessed:

• security_file_permission
• security_mmap_file
• security_path_truncate

kubectl apply -f tracingpolicy.yaml

This TracingPolicy will extract read/write event logs.

Access Sensitive Files from the Pods

Now, exec into the dev-pod and apply the cat command to read /etc/passwd file,

kubectl exec -it nsenter -n default -- cat /etc/passwd

You can verify the logs from the Parseable Console. Next, we will generate an alert in the Parseable to get a notification when a sensitive file is accessed.

Set an Alert in the Parseable

Parseable allows us to generate an alert when the  /etc/shadow file is read from the pod.

• Use the following JSON code to set an alert from the Parseable UI. Click console->config->alert. 

To generate an alert for a specific pod,

{
    "version": "v1",
    "alerts": [
        {
            "name": "Unauthorised access",
            "message":  "/etc/shadow file is accessed by an unauthorised pod",
            "rule": {
                "type": "composite",
                "config": "(process_kprobe_process_arguments =% \"shadow\")"
            },
            "targets": [
                {
                    "type": "webhook",
                    "endpoint": "https://webhook.site/e6a234d3-8bac-4c1b-80df-dda622b3ade2",
                    "skip_tls_check": false,
                    "repeat": {
                        "interval": "10s",
                        "times": 5
                    }
                }
            ]
        }
    ]
}

It will generate an alert when the /etc/shadow file will be accessed from the pod. Set up the target endpoint by going through https://webhook.site and copying your unique URL. Paste it in place of <webhook.site_custom_endpoint>.

• Click Submit button.

Now, let’s verify how the above Parseable setup will trigger an Alert. Again exec into the dev-pod and apply cat command to read  /etc/passwd file and check with the unique URL to see the Alerts,

kubectl exec -it dev-pod -n default -- cat /etc/passwd

• Open the unique URL in a new tab for alert messages.

We can see the message, “unauthorized access triggered in tetrademo” as shown in the above screenshot. This is how the cluster admin can find out the unauthorized access in the cluster and resolve it quickly.

What Next?

We will get a better visualisation in Grafana through the Parseable plugin in our upcoming hands-on lab. 

Grafana Dashboard

We will create a Grafana dashboard to display sensitive file access counts.

Conclusion

In this hands-on lab, We have covered how Parseable fits into the Kubernetes eco-system with Tetragon logs. We have set up a Tetragon logging pipeline with Vector and Parseable. We applied TracingPolicy and generated alerts in Parseable if a sensitive file had been accessed.

References

[1] https://ebpf.io/what-is-ebpf/

[2] https://tetragon.io/docs/

[3] https://www.parseable.io/docs

[4] https://vector.dev/docs/

[5] https://grafana.com/grafana/