Understanding CNI and AWS VPC CNI in EKS

What is CNI?

CNI (Container Network Interface) is a framework that manages networking for containers. It enables Kubernetes pods to communicate with each other and external services by dynamically attaching network interfaces.

In AWS EKS, the Amazon VPC CNI is used by default, allowing pods to get IP addresses from the VPC subnet, ensuring high performance and scalability.

AWS VPC CNI Overview

The AWS VPC CNI (Amazon VPC Container Network Interface) enables Kubernetes pods in EKS to get native VPC IP addresses, ensuring high performance and direct integration with AWS networking.

Key Features

✅ Pods get VPC IPs – Each pod receives an IP from the VPC subnet.
✅ High Performance – Uses AWS networking instead of overlays (no encapsulation).
✅ Elastic IP Allocation – Utilizes secondary ENIs (Elastic Network Interfaces) to scale IPs.
✅ Security & Control – Works with AWS Security Groups & Network ACLs.

How AWS VPC CNI Works

1️⃣ Node Joins Cluster → VPC CNI assigns an Elastic Network Interface (ENI) to the node.
2️⃣ Pods Scheduled → The node assigns secondary IPs from the ENI to the pods.
3️⃣ Pod-to-Pod Communication → Since pods use VPC IPs, they communicate directly without encapsulation.
4️⃣ Pod-to-External Traffic → Uses VPC routing, NAT Gateway, or Internet Gateway for access.

Understanding ENI in AWS VPC CNI

ENI (Elastic Network Interface) is like a network card for AWS EC2 instances (or EKS worker nodes). It allows them to connect to the network and communicate with other resources.

How ENI Works in EKS

🟢 Each worker node in EKS gets one primary ENI.
🟢 The ENI has multiple IPs (one for the node + extra for pods).
🟢 When a pod is created, it gets an IP from the ENI (via AWS VPC CNI).
🟢 Pods communicate directly using these VPC IPs, without extra network layers.

Step-by-Step Breakdown of AWS VPC CNI IP Allocation

1️⃣ A new worker node joins EKS → It gets a main IP + extra IPs.
2️⃣ Pods start running on the node → They take extra IPs from the node’s network interface.
3️⃣ Pods communicate with each other → Since they all have real VPC IPs, they talk directly.
4️⃣ Pods access the internet → If needed, they go through a NAT Gateway or Internet Gateway.

How AWS VPC CNI Assigns ENIs & IPs

1️⃣ A new worker node joins EKS → AWS VPC CNI assigns a primary ENI to the node.

• This ENI has a main IP, which is the node's own IP.

• It also gets extra secondary IPs (based on instance type).
  2️⃣ IPAMD (aws-node DaemonSet) manages these secondary IPs in a warm pool.

• These IPs are pre-allocated for fast pod creation.
  3️⃣ When a pod is scheduled → Kubelet asks CNI Binary for an IP.

• CNI Binary asks IPAMD → IPAMD assigns an IP from the warm pool.

• The IP is attached to the pod and used for communication.
  4️⃣ If more pods need IPs than available →

• IPAMD requests AWS to allocate another ENI.

• New secondary IPs are added to the warm pool.

Components of AWS VPC CNI

Every node has: 1️⃣ CNI Binary → Located inside every worker node (/opt/cni/bin/aws-cni).
2️⃣ IPAMD (aws-node DaemonSet) → Manages IPs for pods across nodes.

Whenever a scheduler tells Kubelet to add a pod: 1️⃣ Kubelet creates the pod and asks the CNI Binary to assign an IP.
2️⃣ CNI Binary requests an IP from IPAMD.
3️⃣ IPAMD assigns an IP from its pool.
4️⃣ CNI Binary attaches the IP to the pod’s network interface.
5️⃣ Pod is now ready to communicate within the cluster & outside! 🚀

How Many ENIs Are Assigned to a Worker Node?

The number of ENIs (Elastic Network Interfaces) assigned to a worker node depends on the EC2 instance type.

ENI & IP Allocation per Instance Type

Example 🔥

🔹 m5.large instance can have 3 ENIs, each supporting 10 IPs.
🔹 Total = (3 × 10) - 1 (for node itself) = 29 IPs available for pods.

So, the number of ENIs increases dynamically as more pods are scheduled, up to the maximum limit of the instance type. 🚀

Dynamic ENI Allocation in AWS VPC CNI

A worker node starts with a single primary ENI. Additional ENIs are attached dynamically based on the number of pods running and the IP address requirements. Here's how:

1️⃣ Primary ENI: When a worker node is launched, it has a primary ENI (eth0) attached by default, providing initial networking capabilities. 2️⃣ Dynamic Attachment of Secondary ENIs: As more pods are scheduled, the Amazon VPC CNI plugin manages the attachment of secondary ENIs (eth1, eth2, etc.) to provide additional IPs for the pods. 3️⃣ IP Address Allocation: Each ENI can have multiple IPs, which are allocated to pods as needed, ensuring efficient resource utilization. 4️⃣ Warm ENI Target: The CNI plugin maintains a warm pool of ENIs and IPs for faster pod startup times. This behavior is controlled by the WARM_ENI_TARGET setting, which defines the number of additional ENIs kept available for immediate use.

Conclusion

AWS VPC CNI is a crucial component of Kubernetes networking in EKS, ensuring efficient pod communication, high performance, and native integration with AWS networking. By understanding ENI allocation, IP management, and how pods communicate, you can optimize your EKS cluster for scalability and reliability.

🚀 Want to explore more about AWS VPC CNI? Drop your thoughts in the comments!

My LinkedIn Profile - https://www.linkedin.com/in/binereet-singh-9a7685316/