The container orchestration standard has been taken over by Kubernetes. However, as it doesn't offer a default observability for security to study the attacks, it raises questions about how organisations should protect their production environment. This issue is resolved with eBPF.
Currently, one of the most popular subjects being addressed in the industry is eBPF. Not only developers that work with the Linux kernel but the cloud-native community is also attracted to it.
- Tetragon is an open-source project of Cilium that offers run-time security, deep observability, and kernel-level transparency using eBPF.
- Parseable is a lightweight, high-performance log analytics stack for developers.
How we will achieve Tetragon logs in the Parseable UI will be the topic of discussion in this hands-on lab. Let's first understand a bit about eBPF before moving further.
What is eBPF?
The extended Berkeley packet filter, or eBPF, allows us to dynamically program the Linux kernel and obtain insights. Generally, eBPF functions in networking, security, and observability.
You can refer to our hands-on lab eBPF- an Overview for more detail.
What is Tetragon?
Tetragon is a powerful Kubernetes-aware security observability and runtime enforcement tool that applies TracingPolicy and filtering directly with eBPF. It ensures the reduction of observation overhead by tracking of any process and real-time policy enforcement.
As shown in the above figure, Tetragon is deployed to the Kubernetes cluster. Tetragon uses hook points in eBPF programs to observe kernel calls. If any TracingPolicy is applied, it filters the logs as per the policy configuration. When any event is filtered using TracingPolicy, it stores Tetragon logs locally or uses a webhook backend like Parseable to collect logs. You can refer to our hands-on lab Tetragon: Empowering Kubernetes Security via Dynamic eBPF Insights for more detail.
Implementation of Tetragon Logs in Parseable through Vector
We'll implement Tetragon logs using TracingPolicy, the Tetragon server stores them in a local file system, and configure Vector, a lightweight, ultra-fast tool to send logs to the Parseable server. At last, we will verify Tetragon logs in Parseable UI with sensitive file access through the Kubernetes pod.
Here is the high-level architecture that shows what we would achieve from this hands-on lab.
Please click on the
LAB-SETUP button to get the Kubernetes cluster ready within a few seconds. We will perform the following tasks to understand how Parseable helps us in auditing Tetragon logs.
- Enable Tetragon in the Kubernetes cluster which stores logs locally.
- Send Tetragon logs in Parseable through Vector.
- Sensitive files will be accessed from the pod then alerts will be generated through Parseable.
We will install Tetragon using Helm.
helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system
Wait for a moment,
kubectl get ds tetragon -n kube-system
- To view the logs,
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f
- To get the events in
better format, install
tetra clifor this.
curl -L https://github.com/cilium/tetragon/releases/latest/download/tetra-linux-amd64.tar.gz | tar -xz
sudo mv tetra /usr/local/bin
- Observe the events with
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact
Extract Tetragon Logs in Parseable via vector
Parseable is a lightweight, high-speed, logging and observability platform for cloud-native applications.
- To install the Parseable in our Kubernetes cluster, run the following script that is attached to the lab setup.
- Wait for a few minutes for the Parseable pod to get started and verify the Parseable pods are running in a parseable namespace.
kubectl get pods -n parseable
Now, the application is ready and exposed to the relevant port (
app-port-31000). We'll now use Vector agent, configure Tetragon logs as its data source, and sink it to the Parseable server.
Follow the installation guide to install Parseable on Kubernetes cluster in detail.
Install and configure Vector Agent
We’ll install Vector through Helm. We have configured a
vector-tetragon-values.yaml file (It is attached to the lab setup) such that the vector can access the
helm repo add vector https://helm.vector.dev
helm install vector vector/vector --namespace vector --create-namespace --values vector-tetragon-values.yaml
It will take a few minutes for the Vector pod to get started. Verify the vector pod is running in a vector namespace.
kubectl get pods -n vector
Now Vector is ready to send the events stored in
/var/run/cilium/tetragon/tetragon.log file to the Parseable tetrademo log stream. Once this is done, you can verify the log events in Parseable UI (Click on
Track Sensitive File Access with Parseable
Although integrating Tetragon logs with Parseable has several advantages, we will concentrate on one of the most notable use cases, namely auditing Tetragon events when accessing sensitive files like
Create a Pod to Access File from the Host
- Create a pod with privileged access Following pod configuration is in lab setup.
kubectl apply -f pod.yaml
- To list the pods,
kubectl get pods
The next step is to apply policy on Tetragon logs so it will extract particular types of logs.
- Apply following TracingPolicy for read/write access. Please open the
tracingpolicy.yamlfile that is attached to the lab setup.
It will monitor a specific file
/etc/passwd. Here, the kprobe is a hook point that hooks on the following kernel functions from where the file can be accessed:
kubectl apply -f tracingpolicy.yaml
This TracingPolicy will extract
read/write event logs.
Access Sensitive Files from the Pods
exec into the
dev-pod and apply the cat command to read
kubectl exec -it nsenter -n default -- cat /etc/passwd
You can verify the logs from the Parseable Console. Next, we will generate an alert in the Parseable to get a notification when a sensitive file is accessed.
Set an Alert in the Parseable
Parseable allows us to generate an alert when the
/etc/shadow file is read from the pod.
- Use the following
JSON codeto set an alert from the Parseable UI. Click
To generate an alert for a specific pod,
It will generate an alert when the
/etc/shadow file will be accessed from the pod. Set up the target endpoint by going through https://webhook.site and copying your unique URL. Paste it in place of
Now, let’s verify how the above Parseable setup will trigger an Alert. Again
exec into the
dev-pod and apply cat command to read
/etc/passwd file and check with the unique URL to see the Alerts,
kubectl exec -it dev-pod -n default -- cat /etc/passwd
- Open the unique URL in a new tab for alert messages.
We can see the message, "unauthorized access triggered in tetrademo" as shown in the above screenshot. This is how the cluster admin can find out the unauthorized access in the cluster and resolve it quickly.
We will get a better visualisation in Grafana through the Parseable plugin in our upcoming hands-on lab.
We will create a Grafana dashboard to display sensitive file access counts.
In this hands-on lab, We have covered how Parseable fits into the Kubernetes eco-system with Tetragon logs. We have set up a Tetragon logging pipeline with Vector and Parseable. We applied TracingPolicy and generated alerts in Parseable if a sensitive file had been accessed.