Deploying a Highly Available MongoDB Cluster on Kubernetes: A Step-by-Step Guide

By Abdul Mueez
Software Engineer | Android & Kubernetes | Building Scalable Mobile & Cloud Solution

---

Introduction

In modern cloud-native applications, ensuring database high availability is critical. As part of my learning journey with Kubernetes, I recently deployed a MongoDB replica set to achieve automatic failover, data replication, and self-healing capabilities.

In this blog, I’ll walk through:
✅ Why MongoDB + Kubernetes?
✅ Step-by-step deployment (with YAML files & commands)
✅ Testing replication & failover
✅ Key lessons learned

---

Why MongoDB Replica Sets? The Heart of High Availability

When downtime costs revenue and data loss is unacceptable, MongoDB’s replica sets deliver enterprise-grade resilience. Here’s how they work in practice:

Fig 1. Client interactions with a MongoDB replica set. Writes route exclusively to the primary, while reads can scale across secondaries.

Key Advantages Demonstrated in This Project

1. Automatic Failover

   • When our primary (mongo-1) failed, Kubernetes automatically:
     ✅ Recreated the pod
     ✅ MongoDB promoted mongo-0 as the new primary
     Zero client-side changes required

2. Data Redundancy

   • Every write to the primary replicated to secondaries via oplog

   • Verified by inserting test documents and querying secondaries:

       kubectl exec -it mongo-0 -- mongosh --eval 'rs.secondaryOk(); db.test.find()'
     

3. Read Scalability

   • Applications can distribute reads to secondaries using:

       spring.data.mongodb.uri=mongodb://mongo-0.mongo,mongo-1.mongo,mongo-2.mongo/db?replicaSet=rs0&readPreference=secondaryPreferred
     

Why Kubernetes Amplifies These Benefits

• StatefulSets ensure each MongoDB pod gets:
  🔹 Stable hostnames (mongo-0, mongo-1, etc.)
  🔹 Dedicated persistent storage

• Headless Service enables direct pod discovery for replication

Real-World Impact: This architecture survived simulated failures with no data loss or manual intervention – exactly what production systems demand.

Why MongoDB Replica Sets on Kubernetes?

MongoDB’s replica sets provide:

• Automatic failover (if the primary node fails, a secondary takes over)

• Data redundancy (asynchronous replication via oplog)

• Read scalability (distribute queries across secondaries)

Kubernetes StatefulSets complement this by:

• Providing stable hostnames (mongo-0, mongo-1, etc.)

• Managing persistent storage per pod

• Ensuring ordered pod initialization (critical for replica sets)

---

Step 1: Setting Up the MongoDB StatefulSet

Key Components

1. StatefulSet → Manages MongoDB pods with unique identities

2. Headless Service → Enables direct pod-to-pod communication

YAML Configuration

mongo-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo
spec:
  serviceName: "mongo"
  replicas: 3
  selector:
    matchLabels:
      app: mongo
  template:
    metadata:
      labels:
        app: mongo
    spec:
      containers:
      - name: mongo
        image: mongo:6.0
        command: ["mongod", "--replSet", "rs0", "--bind_ip_all"]
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: mongo-persistent-storage
          mountPath: /data/db
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "standard"
      resources:
        requests:
          storage: 1Gi

mongo-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: mongo
spec:
  clusterIP: None  # Headless mode
  ports:
  - port: 27017
  selector:
    app: mongo

Deployment:

kubectl apply -f mongo-statefulset.yaml
kubectl apply -f mongo-service.yaml

Output:

All pods (mongo-0, mongo-1, mongo-2) running successfully.

---

Step 2: Initializing the Replica Set

Once pods are ready, initialize the replica set:

kubectl exec -it mongo-0 -- mongosh --eval 'rs.initiate({
  _id: "rs0",
  members: [
    {_id: 0, host: "mongo-0.mongo:27017"},
    {_id: 1, host: "mongo-1.mongo:27017"},
    {_id: 2, host: "mongo-2.mongo:27017"}
  ]
})'

Verify Status:

kubectl exec -it mongo-0 -- mongosh --eval 'rs.status()'

Output:

mongo-1 is PRIMARY; others are SECONDARY.

---

Step 3: Testing Replication

Insert Data into Primary

kubectl exec -it mongo-1 -- mongosh --eval 'db.test.insertOne({message: "Hello from primary!"})'

Read from Secondaries

kubectl exec -it mongo-0 -- mongosh --eval 'rs.secondaryOk(); db.test.find()'
kubectl exec -it mongo-2 -- mongosh --eval 'rs.secondaryOk(); db.test.find()'

Output:

Data successfully replicated to all nodes.

---

Step 4: Simulating Failover

Delete Primary Pod

kubectl delete pod mongo-1

Verify New Primary Election

kubectl exec -it mongo-0 -- mongosh --eval 'rs.status()'

Output:

mongo-0 promoted to PRIMARY; mongo-1 recreated as secondary.

---

Lessons Learned

1. StatefulSets are mandatory for stateful apps like MongoDB.

2. Headless Services enable direct pod DNS resolution.

3. Testing failover is critical to validate high availability.

4. Production considerations:

   • Use anti-affinity rules to prevent co-located pods.

   • Choose cloud-native storage (e.g., EBS, Azure Disk).

---

Final Thoughts

This project demonstrated how Kubernetes StatefulSets and MongoDB replica sets work together to deliver a resilient, scalable database layer. The combination is production-ready with minor adjustments.