Cilium is a cloud-native networking and security project that brings the power of eBPF (extended Berkeley Packet Filter) to Kubernetes. Traditional Kubernetes networking solutions rely heavily on iptables, IP routing, and overlay tunnels (like Flannel or Calico in VXLAN mode). These approaches work but introduce scaling and performance limitations as clusters grow.

Cilium rethinks networking: instead of managing long iptables chains, it uses eBPF programs directly in the Linux kernel to handle packet forwarding, load balancing, and security. This results in faster networking, reduced overhead, and fine-grained security policies that scale effortlessly in large clusters.

In this article, we’ll explore:

IP Address Management (IPAM) in Cilium
Routing Modes: Encapsulation vs Native
Endpoints and Identities in Cilium
Packet Walk: How traffic flows from Pod to Pod

1. IP Address Management (IPAM) in Cilium

Every Pod in Kubernetes needs a unique IP address. This address assignment process is handled by IPAM (IP Address Management). Without correct IPAM, Pods would collide on IPs, and networking between nodes would fail.

How Pod CIDRs Work

When you create a Kubernetes cluster, you usually define a Pod CIDR range. For example:

--cluster-cidr=10.244.0.0/16

This range defines all possible Pod IPs in the cluster. Kubernetes then divides this CIDR across nodes. For a 3-node cluster:

Node1 → 10.244.1.0/24
Node2 → 10.244.2.0/24
Node3 → 10.244.3.0/24

Whenever a Pod is scheduled on Node1, it receives an IP from 10.244.1.0/24. This ensures no overlap between nodes.

IPAM Modes in Cilium

Cilium supports multiple modes for assigning Pod IPs:

🔹 Kubernetes Host-Scope IPAM

Kubernetes itself allocates Pod CIDRs to each node.
Controlled by kube-controller-manager flags:

--allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16 --node-cidr-mask-size=24

To check Pod CIDR on a node:

kubectl get node <node-name> -o yaml | grep podCIDR

🔹 Cluster-Pool IPAM (Default in Cilium)

The Cilium operator itself assigns Pod CIDRs, not Kubernetes.
You configure a pool of available ranges, and Cilium dynamically assigns them.
Example Helm config:

🔹 Cloud-Specific IPAM

Cilium also integrates with cloud providers:

AWS ENI: Pods get IPs directly from VPC subnets.
Azure IPAM: Allocates IPs from Azure virtual networks.
GKE IPAM: Uses Google VPC-native IPAM.

These modes remove overlays entirely and integrate Pods directly into cloud VPC routing.

2. Routing Modes in Cilium

Once Pods have IPs, the next question is: how do Pods on different nodes communicate?

Cilium provides two main routing modes:

🔹 Encapsulation Mode (Default)

In Encapsulation mode, Pod-to-Pod traffic across nodes is wrapped (encapsulated) in another packet so that it can traverse the underlying data center network.

Scenario:

Pod1 on Node1 (10.244.1.2) wants to talk to Pod2 on Node2 (10.244.2.3).
These Pod IPs are not routable on the physical network.
Normally, a router would drop such packets.

How Encapsulation solves this:

Cilium wraps (encapsulates) the Pod-to-Pod packet inside another packet.
Outer packet source/destination = Node1 IP ↔ Node2 IP.
Physical network routes it successfully.
On arrival, Node2 decapsulates and delivers it to Pod2.

Key Points:

Uses tunneling protocols (VXLAN, Geneve).
Requires node-to-node connectivity.
Firewalls must allow tunneling traffic.
Adds ~50 bytes of overhead (encapsulation headers).

🔹 Native Routing Mode

Instead of encapsulating traffic, Cilium can use routing protocols like BGP to advertise Pod CIDRs directly to the network.

How it works:

Each Cilium agent runs on every node.
The agent advertises the node’s Pod CIDR to the router using BGP.
Routers update their routing tables.
Packets between Pods flow natively without encapsulation.

Advantages:

No encapsulation overhead.
Better performance at scale.
Fully routable Pod IPs across the data center.

Requirements:

A network fabric that supports BGP.
Cluster admins must coordinate with network team to configure routers.

3. Endpoints and Identities in Cilium

🔹 Endpoints

In Cilium, an Endpoint represents a Pod (or container) in the cluster.

Every Pod = Endpoint.
Endpoints are tracked by Cilium with unique IDs.

🔹 Identities

Cilium introduces a label-based security model.

Every Pod has labels (e.g., app=frontend, env=prod).
Cilium converts unique label sets into identities.
Each identity gets a numeric ID (e.g., ID 12345).

This means network policies are applied based on identities, not IPs.

Why this matters:

Pod IPs are ephemeral (Pods restart, get new IPs).
Labels are stable (a frontend Pod is always app=frontend).
Policies remain consistent regardless of IP changes.

🔹 Endpoint ↔ Identity Relationship

Every Endpoint (Pod) is linked to an Identity.
If labels change, Cilium updates the Identity dynamically.
Policies are enforced between Identities, not raw IPs.

This provides scalable, label-driven security.

4. Packet Walk in Cilium: Pod-to-Pod

Now let’s put it all together with a step-by-step packet walk.

Traffic flow from Pod A → Pod B (across nodes):

Pod sends packet → destined for ServiceIP or PodIP.
Packet exits Pod’s veth → hits host-side veth interface.
From-container BPF program executes:
- Performs load balancing (if Service IP).
- Updates conntrack.
- Rewrites MACs.
Packet enters kernel routing → determines next hop.
On NIC egress, to-netdev BPF program runs:
- Minimal validation.
- Sends packet to data center network.
Data center fabric routes packet to destination node (via encapsulation or BGP).
On destination node ingress, from-netdev BPF program executes:
- Identifies target Endpoint.
- Calls Pod ingress BPF program.
To-container BPF program applies policy checks.
If allowed, packet is delivered to Pod’s eth0.

The return path follows the same sequence in reverse.

🔑 Key Insights

Cilium IPAM decides how Pods get IPs: Kubernetes vs Cluster-pool vs Cloud-native IPAM.
Routing modes define Pod communication:
- Encapsulation → Simple, portable, but overhead.
- Native routing (BGP) → High-performance, needs external network integration.
Cilium Endpoints & Identities provide dynamic, label-based security instead of IP-based rules.
Packet Walk shows how eBPF intercepts packets at multiple stages, handling load balancing, policy, and forwarding.

Exploring Cilium: Deep Dive into IPAM, Routing Modes, Identities, and Packet Walk