Understanding Pod Deletion in Kubernetes: Deployments vs. StatefulSets

---

Ever scaled down a Deployment and wondered which pod gets the axe first? Kubernetes doesn't play dice with your pods (well, not until the very end). It actually follows a clear hierarchy to decide which pod to delete when scaling down a Deployment or ReplicaSet. ⚖️

Let me break it down in my own words. This is the stuff I wish I knew earlier.

---

How Pod Deletion Works in Deployments/ReplicaSets

Here’s the step-by-step logic K8s follows:

1. Pending Pods Go First

Pods stuck in Pending state — maybe they’re waiting on resources or unschedulable — get removed first. They're dead weight anyway.

2. pod-deletion-cost Annotation

Want more control? Use this annotation: controller.kubernetes.io/pod-deletion-cost

• Lower value = deleted first

• Higher value = retained longer

Super useful if you're optimizing for locality (e.g., stick pods to a region or zone).

3. Nodes with More Replicas

When deciding among healthy pods, K8s prefers deleting pods from nodes that are hosting more replicas. It's a balancing act.

4. Older Pods Before Newer

If it still has to choose, it picks the older pods first. Makes sense — older pods might be:

• Using more memory/CPU

• Stale in config or state

• Caching too much

5. Random (Fallback Only)

If everything above is equal, then it finally picks randomly. But that’s a last resort.

---

🔄 What Really Happens When You kubectl delete

Ever wondered what happens behind the scenes when you run:

kubectl delete

When you delete a resource like a ReplicaSet or Deployment, Kubernetes uses a propagation policy to determine what happens to its dependent resources — typically Pods.

The three strategies are:

1. Foreground
   The parent (e.g., ReplicaSet) enters a "deletion in progress" state.
   → Pods are deleted first
   → Then the parent is removed.

2. Background
   The parent is deleted immediately.
   → Garbage Collector deletes the dependent Pods in the background.

3. Orphan
   The parent is deleted right away, but dependent Pods are left behind (orphaned).
   → Useful if you want to preserve running Pods for inspection or reuse.

These policies affect how cascading deletion behaves — especially important when scripting cleanup or debugging.

---

🔐 Why This Matters in Production

These rules aren’t just trivia — they matter:

• They help avoid disruption by keeping fresher and more stable pods around.

• They let you influence behavior with annotations (lifesaver in complex systems).

• They help with safe scale-down, especially for web apps or APIs.

---

But What About StatefulSets?

Stateful workloads — like PostgreSQL, Kafka, Cassandra — play by different rules. And for good reason: data.

StatefulSet Scale Down Strategy

When you scale down a StatefulSet, K8s doesn’t check cost or age. It just removes pods in reverse ordinal order:

• First: pod-N

• Then: pod-(N-1)

• Finally: pod-0

Always highest-numbered pod first. Think of it like peeling layers off an onion.

Why This Matters?

StatefulSets provide key guarantees that are essential for stateful workloads:

• Stable network identity: Each pod gets a persistent DNS entry, such as postgres-0.postgres.svc.cluster.local, which doesn't change even after restarts. This stability is critical for distributed systems that rely on consistent endpoints.

• Persistent storage: Each pod is bound to its own PersistentVolumeClaim (PVC). The data is not shared between pods, and it remains intact across restarts or rescheduling, ensuring data durability.

• Ordered and deterministic operations: Pods are created and deleted in a specific sequence. This order is crucial when pods have different roles (e.g., primary vs. replica databases) or depend on each other during startup or shutdown.

You don’t want K8s killing your DB primary before the replicas are even ready.

---

🧪 Give It a Go: Hands-On Pod Deletion in Action

All this theory is cool — but let’s actually try it out. You don’t need anything fancy — just a working cluster and kubectl.

🔁 Deployment Scale Down (Smart & Context-Aware)

# Step 1: Create a simple Deployment with 3 replicas
kubectl create deployment nginx-deploy --image=nginx
kubectl scale deployment nginx-deploy --replicas=3

# Optional: Annotate a pod with pod-deletion-cost
kubectl annotate pod <pod-name> controller.kubernetes.io/pod-deletion-cost="100"

# Step 2: Scale down the Deployment
kubectl scale deployment nginx-deploy --replicas=1

What to watch:

• Pending pods (if any) should go first.

• If annotations are present, they influence the order.

• Older pods tend to go before newer ones.

kubectl get pods -o wide --sort-by=.metadata.creationTimestamp

---

StatefulSet Scale Down (PostgreSQL Example)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: "postgres"
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:13
        ports:
        - containerPort: 5432
          name: postgres
        env:
        - name: POSTGRES_PASSWORD
          value: "example"
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: postgres-storage
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

# Step 1: Apply the StatefulSet YAML above
kubectl apply -f postgres-statefulset.yaml

# Step 2: Scale down to 1 replica
kubectl scale statefulset postgres --replicas=1

What you’ll notice:

• K8s will remove postgres-2, then postgres-1

• postgres-0 — the likely "primary" — stays till the end

List pods to watch deletion order:

kubectl get pods -l app=postgres

This hands-on test makes all the deletion rules much more intuitive — and you’ll remember them better after trying it.

---

🧐 TL;DR - Deletion Logic Summary

---

💬 Final Thoughts

This is one of those behind-the-scenes behaviors in Kubernetes that’s easy to overlook but super important when you’re dealing with production workloads. Especially with critical services, knowing how K8s makes these decisions can save you from big headaches.

Hope this cleared things up!

If you liked this breakdown, let’s connect: ✨
📎 Follow me on LinkedIn or X
✍️ Got questions? Drop a comment — always happy to write a follow-up!

Until next time — may your replicas scale smoothly and your primaries live long. 🚀

#Kubernetes #StatefulSet #PodDeletion #CloudNative #DevOps #K8sTips #LearningInPublic

---