Inspektor Gadget Explained !

observability is very important in kubernetes and containerized environments, and debugging and troubleshooting kubernetes is always hard, tools that are effective and efficient are always needed, so unlike traditional systems where if something goes wrong you can easily ssh into the server and inspect, in kubernetes enviroments its a bit complex because of so many objects like pods, services and deployments that are running and interacting at the same time, so whenever something goes wrong or if there is a performance bottleneck or a misconfiguration it becomes difficult to pinpont the root cause, eBPF technology changes everything at the root level, it allows programs to be run in a isolated sandboxed enviroment securely and safely just as it is the kernel code itself, inspektor gadget is powerd by eBPF and kubernetes awareness of inspektor gadget makes it even more special. not only it provides deep visibility into low level system events like syscalls, network traffic, and file access but it also associates those events with high level kubernetes concepts, In this blog, we’ll explore how inspektor gadget works, the challenges it solves, and walk through hands on examples to help you get started. If you're ready to unlock powerful insights into your kubernetes and linux environments, let’s dive in!

What is Inspektor Gadget ?

Inspektor Gadget is a collection of tools, known as gadgets, that enable developers to inspect kubernetes and linux systems using eBPF programs in an accessible way. It also serves as a framework, providing a method for eBPF developers to easily build, package, deploy, and run these gadgets. By enriching kernel level data with higher level kubernetes concepts, inspektor gadget supercharges eBPF programs, making them easy for anyone to use and understand.

Inspektor Gadget’s Capabilities

• With inspektor gadget, you can create and package eBPF programs into reusable OCI images, or gadgets, that are easy to share and deploy. Think of these gadgets as your observability toolkit, portable, lightweight, and prepared to take on specific tasks like monitoring network activity or tracing file operations.

• Whether you're managing a standalone linux server or a kubernetes cluster, inspektor gadget has you covered because it's designed to integrate seamlessly with both environments, allowing you to inspect systems wherever your workloads run.

• You can collect system data with a single command, then send it to your preferred observability tools, such as prometheus or grafana. Soon, we will be able to do this declaratively, which will streamline the process of automating and scaling your monitoring workflows.

• Inspektor gadget lets you lock down operations to prevent unauthorized access and to prioritize safety with built in mechanisms to control which gadget to run, This makes sure that the security of our system is not jeopardised by oour observability attempts.

• One of the most powerful features of inspektor gadgets is automatic enrichment. To put it simply, the data that eBPF collects from the kernel is unaware of kubernetes, container runtimes, or any other high level userspace concepts. Thus, inspektor gadget automatically uses kernel primitives like mount namespaces, pids, or similar to figure out which high level concepts they relate to, such as kubernetes pods, container names, dns names, etc. The process of enhancing the eBPF data with these high level concepts is termed as enrichment.

• Webassembly is also supported to process your data your way, thanks to inspektor gadget's support for WASM modules this allows us to write custom opertors in wasm supported language, the flexibility lets you customize data processing to your specific needs , whether you are transforming outputs or filtering events.

• Inspektor gadget even supports multiple operations modes , you can use it via a CLI or setup client server model or interact through an API or even embed it directly to your GO applciations as a library, its very convenient and specifically designed to fit seamlessly into your toolkit.

A Little Bit About eBPF

eBPF (Extended Berkeley Packet Filter) is a technology that lets you run sandboxed programs inside the linux kernel. It’s like giving your system a superpower, capturing events like file accesses or network packets without modifying kernel code. eBPF is fast, safe, and versatile, powering modern observability and security tools. However, writing eBPF programs can be daunting, requiring kernel expertise. Inspektor gadget solves this by providing gadgets, abstracting the complexity while delivering eBPF’s full potential.

Setting Up and Using Inspektor Gadget on Kubernetes

Now that we have explored inspektor gadget and its capabilities, let’s get our hands dirty with some hands on learning, my favorite part!

Inspektor gadget offers a variety of gadgets for observability, and there are multiple ways to use it on kubernetes and linux systems. In this section, I’ll demonstrate how to troubleshoot and debug a kubernetes cluster using two gadgets trace_open and traceloop Inspektor gadget can be used in two primary ways, on kubernetes clusters via kubectl plugins or on standalone linux systems using the ig cli.

For this demo, i’ll focus on the kubernetes approach, which involves a kubectl plugin running on your system and a DaemonSet deployed in the cluster to collect data.

My kubernetes cluster gadget-kluster with two nodes, is deployed on GKE in the us-west1-a region. You must have a kubernetes cluster as well, for local development you can use Minikube or Kind.

Krew is the recommended way to install inspektor gadget. To install krew, you must have git installed in your system. Use the git version command to check if git is installed on your system. If it is not installed, you must install git first. You can refer to git-scm.com for installation instructions.

Run this command in your bash or zsh shell to download and install krew plugin manager for kubectl

(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)

An output like this is expected when krew is installed successfully. If you encounter any error, try repeating the process in a fresh environment or check whether the required versions are compatible.

Run this command to add krew to your PATH environment variable

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

To check if krew is successfully installed, run the kubectl krew command. If you see output similar to this then Krew has been installed successfully.

Now that krew is successfully installed, we can run the command kubectl krew search to get a list of all available kubectl plugins that can be installed via Krew. I simply use grep to search for the inspektor gadget plugin. As we can see, the plugin is not currently installed, so we need to install it.

Run kubectl krew install gadget to install the inspektor gadget kubectl plugin. After the command executes successfully, if we run kubectl krew search | grep gadget again, we can see that the plugin is now installed, it will show yes in the INSTALLED column.

Run kubectl gadget deploy to deploy the inspektor gadget daemon into your kubernetes cluster. The gadget runs as a DaemonSet on the cluster nodes to collect and analyze system data, so it is very important to deploy it first.

Once inspektor gadget is successfully deployed, if we run kubectl get ns, we can see that a new namespace called gadget has been created for inspektor gadget. All related resources will be deployed in this namespace.

Run kubectl get all -n gadget to view all resources inside the gadget namespace. You will see that the pods and daemon sets are up and running, which indicates that the deployment of inspektor gadget was successful.

trace_open is one of the gadget provided by inspektor gadget that traces file open events across containers in your kubernetes cluster. It emits events whenever a file is opened inside a pod.

Before using trace_open, we need to run an application that generates file access events, so the gadget has something to trace. Run the following command to start a simple BusyBox pod:

kubectl run --restart=Never --image=busybox mypod -- sh -c 'while /bin/true ; do whoami ; sleep 3 ; done'

This command launches a BusyBox pod that prints the current user every 3 seconds. These repeated actions help generate file open events, enables the trace_open gadget to capture and display meaningful output.

The trace_open gadget allows you to trace file open events within containers. By using the --podname flag, you can target a specific pod, in this case, mypod without specifying a node. This enables the gadget to identify and trace the pod across all nodes in the cluster.

To begin tracing file access events for mypod, run the following command:

kubectl gadget run trace_open:latest --podname mypod

This command invokes the latest version of the trace_open gadget and attaches it to the pod named mypod. It uses eBPF to trace system calls related to file access and emits events each time a file is opened by a process inside the pod.

In this example, the mypod container runs a BusyBox shell loop that prints the current user every 3 seconds. This behavior results in repeated file accesses (e.g. reading /etc/passwd), which will be traced and printed by the trace_open gadget in real time.

The whoami command appears to open /etc/passwd in order to resolve the user ID to a corresponding username. This file access is captured by the trace_open gadget.

You can stop the gadget at any time by pressing Ctrl+C in the terminal.

This example shows the capabilities of inspektor gadget specifically the trace_open gadget, in providing low level observability into container workloads. The detailed output generated during this trace is shown in the image.

let’s explore the traceloop gadget, also referred as a system call flight recorder. This nickname makes perfect sense because just like a flight recorder or "black box" in an airplane captures every detail of a flight for later analysis in case of an incident, traceloop records system calls made by containers or processes in real time, storing a detailed trace that you can go through to understand what happened leading up to an issue. System calls are the fundamental interactions between a program and the Linux kernel (e.g. opening files, sending network packets), and traceloop captures these interactions with precision, making it an invaluable tool for debugging crashes, performance issues and unexpected behavior.

To use the traceloop gadget, we need to generate some events by running an application. Let’s create a dedicated namespace for our traceloop resources to keep things organized

kubectl create ns test-traceloop-ns

To capture system calls and analyze their output for insights into system behavior, let’s run an application that generates events. Use the following command to create and run a Busybox pod in the test-traceloop-ns namespace:

kubectl run -n test-traceloop-ns test-traceloop-pod --image=busybox --command -- sleep infinity

This command deploys a pod named test-traceloop-pod that runs a container with the sleep infinity command, keeping the container active indefinitely.

To deploy and run the traceloop gadget, execute the following command:

kubectl gadget run traceloop:latest --namespace test-traceloop-ns

This deploys the traceloop gadget in the test-traceloop-ns namespace, where it runs in the background as a system call flight recorder. It quietly captures system calls using eBPF and stores them in a ring buffer for later analysis, even for containers that have terminated. You can view the captured traces at any time using kubectl gadget traceloop show.

To generate system call data, run the following command to open an interactive shell inside the pod:

kubectl exec -ti -n test-traceloop-ns test-traceloop-pod -- /bin/hush

Then, inside the shell, type:

ls

This executes the ls command, which invokes system calls that the traceloop gadget will record. Since traceloop runs in the background, it won’t display any data until you stop it by pressing Ctrl+C. This will terminate the gadget and show the recorded output.

When I stop the traceloop gadget by pressing Ctrl+C it displays output with fields such as K8S.NODE K8S.NAMESPACE K8S.PODNAME K8S.CONTAINERNAME CPU PID COMM SYSCALL PARAMETERS and RET these fields provide detailed information about the system calls made by containers in our kubernetes cluster, enriched with Kubernetes metadata for easier correlation.

Here’s how traceloop works: it uses eBPF to silently capture system calls in the background. When I stop the tracing session with Ctrl+C it presents the recorded data. This data is invaluable for troubleshooting and debugging our kubernetes clusters, as well as enhancing observability by offering low level insights into container behavior.

conclusion

The powerful eBPF based features and smart enrichment of inspektor gadget are innovative for kubernetes and linux observability. See the inspektor gadget documentation to learn about other gadgets and features. Tools like inspektor gadget will become even more essential as the cloud native environment grows. Try it out, explore around with its features, and discover how it could boost your kubernetes workflows. In this blog post, we tried out two gadgets. I plan to explore inspektor gadget in more depth and explain more of its features in future blog posts.

I'm excited about this project. Thanks for reading, and stay tuned for more such blogs!