A Beginner's Guide to StatefulSets: Ensuring Data Persistence in Kubernetes

Introduction

When deploying stateful applications such as databases in Kubernetes, ensuring data persistence and maintaining a stable pod identity is critical. Unlike stateless applications, stateful apps require both persistent storage and predictable scaling behavior. Kubernetes provides StatefulSets to effectively manage these applications. This guide will explore how StatefulSets interact with StorageClasses and Persistent Volume Claims (PVCs), ensuring that data remains safe and accessible, even when pods are rescheduled or deleted.

---

What Are StatefulSets?

StatefulSets are a type of Kubernetes resource that manage stateful applications. These applications need to maintain certain characteristics across pod restarts and scaling operations, such as:

Identity and Stable Network hostnames: StatefulSets are used for those applications that require stable network identity and hostnames. Whenever the pod is created, it gets a unique name, an ordinal index appended to its name. For example, the pod name looks like StatefulSet Pod-0, StatefulSet Pod-1, StatefulSet Pod-2, and so on.

• Ordered Deployment and Scaling: StatefulSets deploy pods sequentially, meaning each pod is created only after the previous one is successfully created. In contrast, Deployments create all pods at once. When deleting pods, StatefulSets follow reverse order — the newest pod is deleted first.

• Data Persistence:StatefulSets are used for applications that need to keep data, like databases. They allow permanent volumes to be attached and mounted to the pods. So, if a pod is rescheduled or restarted, it will still have all its data.

• Headless Services: A powerful feature of StatefulSets is the ability to use Headless Services, which provide DNS records for each pod’s hostname. This enables direct communication with specific pods, making it easier to manage stateful applications.

StatefulSets vs Deployments:

• StatefulSets: Provide stable identities and persistent storage. Ideal for applications like databases (e.g., MongoDB, MySQL).

• Deployments: Used for stateless applications, where stable identities and persistent storage aren’t necessary.

• Scalability and Rolling Updates: Deployments are typically used for stateless applications. They allow you to scale your application with replicas and ensure no downtime during updates.

• No Stable Hostnames: Deployments don't provide stable hostnames, meaning each pod gets a randomly generated name when it's created.

• Load Balancing: Deployments work with Services, which handle load balancing. This helps distribute traffic across multiple pods, making the application more available and reliable.

---

StatefulSet and Headless Service

A StatefulSet in Kubernetes allows pods to have stable, unique identities, which is important for stateful applications. If you want the pods in a StatefulSet to have stable DNS names for communication, you can use a Headless Service.

Key Points:

• Stable DNS Names: By using a Headless Service, each pod in the StatefulSet can be accessed by a unique DNS name (e.g., my-app-0.my-app-service).

• Headless Service: A Service with clusterIP: None makes it headless, allowing each pod to have a unique DNS entry.

Example: Headless Service YAML

apiVersion: v1
kind: Service
metadata:
  name: my-app-service  # Name matches `serviceName` in StatefulSet
spec:
  clusterIP: None  # Makes it a Headless Service
  selector:
    app: my-app
  ports:
    - port: 27017
      name: mongo

Example: StatefulSet YAML (with serviceName)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-app
spec:
  serviceName: "my-app-service"  # Matches the Service name
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: mongo
          image: mongo
          volumeMounts:
            - name: storage
              mountPath: /data/db
  volumeClaimTemplates:
    - metadata:
        name: storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: gp2

How It Works:

• DNS Name Resolution: With the serviceName set to my-app-service, the StatefulSet Pods can be accessed through stable DNS names like my-app-0.my-app-service, my-app-1.my-app-service, and so on.

• Pod-to-Pod Communication: This is useful for applications where each pod needs to be identified individually (e.g., for database replication).

---

Understanding volumeClaimTemplates

In a StatefulSet, each pod requires its own Persistent Volume (PV). Manually creating a PVC for each pod would be cumbersome, especially if the number of pods is dynamic. To automate this process, Kubernetes uses volumeClaimTemplates.

How It Works:

• volumeClaimTemplates automatically creates a Persistent Volume Claim (PVC) for each pod in the StatefulSet.

• If a StorageClass is specified, Kubernetes automatically provisions the Persistent Volume (PV) and binds it to the PVC.

• The PVC is then mounted as a volume inside the pod, allowing the application to store data persistently.

YAML Example:

Here’s a YAML example of a StatefulSet with volumeClaimTemplates for automatic PVC creation:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-stateful-app
spec:
  serviceName: "my-stateful-service"
  replicas: 3
  selector:
    matchLabels:
      app: my-stateful-app
  template:
    metadata:
      labels:
        app: my-stateful-app
    spec:
      containers:
        - name: mongo
          image: mongo
          volumeMounts:
            - name: storage
              mountPath: /data/db
  volumeClaimTemplates:
    - metadata:
        name: storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: gp2

Key Points:

StatefulSets ensure that each replica gets its own unique Persistent Volume (PV). If you specify 3 replicas, Kubernetes will create 3 separate volumes, one for each pod, using the volumeClaimTemplates.

Here’s how it works:

1. Unique Volumes for Each Replica: For a StatefulSet with 3 replicas, Kubernetes will create 3 Persistent Volume Claims (PVCs), each bound to a separate Persistent Volume (PV).
   Example:

   • Pod my-app-0 → PVC my-app-0-storage → PV (Volume for my-app-0)

   • Pod my-app-1 → PVC my-app-1-storage → PV (Volume for my-app-1)

   • Pod my-app-2 → PVC my-app-2-storage → PV (Volume for my-app-2)

2. VolumeClaimTemplate in StatefulSet: This ensures each pod has its own persistent storage. The volumes are not shared between pods to maintain isolation and consistency.

3. Data Persistence: If a pod is rescheduled or restarted, it will always reattach to its respective volume, ensuring no data loss.

---

Where Is the Data Stored?

In a StatefulSet, each pod has its own persistent storage. The data is stored on a Persistent Volume (PV), which is bound to the Persistent Volume Claim (PVC). The data location depends on the storage backend defined in the StorageClass.

For example:

• If you’re using AWS EBS with the gp2 StorageClass, the PV is an EBS volume, and the data is stored there.

• If the pod moves to a different node (due to rescheduling), the EBS volume will be detached from the old node and attached to the new node, ensuring data availability.

---

Non-Cascading vs Cascading Deletion

When managing StatefulSets, Kubernetes offers two deletion strategies: cascading deletion and non-cascading deletion.

1. Cascading Deletion:

   • Pods and StatefulSet are both deleted when the StatefulSet resource is deleted.

   • Data associated with the pods may also be deleted, depending on your storage settings.

2. Non-Cascading Deletion:

   • Only the StatefulSet is deleted, but the pods remain running.

   • This is useful when you want to keep the pods running while cleaning up the StatefulSet definition itself.

Example Command for Non-Cascading Deletion:

To delete a StatefulSet without deleting its pods, use the following command:

kubectl delete statefulset <statefulset-name> --cascade=orphan

This will orphan the pods, leaving them running even after the StatefulSet is deleted.

---

How Kubernetes Manages Storage (In Kubeadm and AWS)

In AWS (EKS):

When you define a StorageClass like gp2, Kubernetes will automatically provision an EBS volume and bind it to the PVC. The data is stored on the EBS volume and persists even if the pod is rescheduled or terminated.

In Kubeadm:

When using manual StorageClasses in Kubeadm (for on-prem Kubernetes clusters), the Persistent Volume (PV) is created automatically based on the PVC. However, for storage types like hostPath, you must manually define the path in the PV to link it to the actual storage location.

Example (HostPath PV in Kubeadm):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: manual-pv
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /mnt/data

Without manually providing the correct path for hostPath or other local storage types, the PVC will remain in a Pending state.

---

Conclusion

Kubernetes StatefulSets offer a robust solution for managing stateful applications that need both persistent storage and stable identities. By utilizing volumeClaimTemplates, StatefulSets can automatically create and manage Persistent Volume Claims (PVCs) for each pod, ensuring that data is preserved across pod restarts and rescheduling. Whether you are working with cloud providers like AWS (EKS) or managing an on-prem Kubernetes cluster with Kubeadm, understanding how StatefulSets and storage resources work together is key to managing stateful applications in a scalable and reliable way.