How to Set Up Kafka and ZooKeeper with High Availability on AWS ECS

Let’s now dive into a practical deployment guide for running Kafka and ZooKeeper in a containerized, HA-ready setup using AWS ECS (EC2 or Fargate).

📦 Architecture Overview

We will deploy:

• Apache ZooKeeper cluster (3 nodes for HA quorum)

• Apache Kafka brokers (3 brokers for replication and load balancing)

• Each broker runs in its own ECS task

• Internal networking via ECS Service Discovery

• Data persistence with Amazon EBS (EC2) or EFS (Fargate)

                +----------------------+
                |    Kafka Clients     |
                +----------------------+
                          |
              ┌───────────▼────────────┐
              │    AWS ECS Cluster     │
              │   (Kafka + ZooKeeper)  │
              └────────────────────────┘
     +-------------------+    +-------------------+
     |  Kafka Broker 1   |    |  Kafka Broker 2   |
     | zookeeper:2181    |    | zookeeper:2181    |
     +-------------------+    +-------------------+
              |                      |
     +-------------------+    +-------------------+
     | ZooKeeper Node 1  |    | ZooKeeper Node 2  |
     +-------------------+    +-------------------+

---

🧱 Step-by-Step: Kafka & ZooKeeper on ECS

---

1. 📁 Containerize Kafka and ZooKeeper

Use Bitnami or Confluent Docker images (or build your own):

# docker-compose.yml (for local testing)
version: '3'
services:
  zookeeper:
    image: bitnami/zookeeper:3.9
    ports:
      - "2181:2181"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_MY_ID=1
      - ZOO_SERVERS=server.1=zookeeper:2888:3888

  kafka:
    image: bitnami/kafka:3.6
    ports:
      - "9092:9092"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - ALLOW_PLAINTEXT_LISTENER=yes

Push these images to Amazon ECR.

---

2. 🛠️ Create ECS Cluster (Fargate or EC2)

• Use ECS Console or IaC (Terraform/CloudFormation)

• For HA, prefer EC2 + Auto Scaling Group

• Ensure subnets span multiple AZs

• Attach ECS instances to a shared security group

---

3. 🔐 Set Up IAM, Security Groups & Networking

• ECS task role with permissions for:

  • ECR pull

  • CloudWatch Logs

• Security Groups:

  • Kafka → allow ports 9092

  • ZooKeeper → allow ports 2181, 2888, 3888

• Enable ECS Service Discovery (via AWS Cloud Map or Route 53)

---

4. 🚢 Deploy ECS Services

Deploy each Kafka and ZooKeeper instance as a separate ECS service:

• 🟠 ZooKeeper:

  • Desired count: 3 (for quorum)

  • Static DNS names via ECS Cloud Map (zookeeper1.service.local, etc.)

• 🔵 Kafka:

  • Desired count: 3

  • Each configured with:

    • KAFKA_CFG_BROKER_ID

    • KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper1.service.local:2181,...

Sample ECS Task Definition Snippet (Kafka)

{
  "containerDefinitions": [
    {
      "name": "kafka-broker",
      "image": "your_account_id.dkr.ecr.region.amazonaws.com/kafka:latest",
      "essential": true,
      "portMappings": [
        { "containerPort": 9092, "hostPort": 9092 }
      ],
      "environment": [
        { "name": "KAFKA_CFG_BROKER_ID", "value": "1" },
        { "name": "KAFKA_CFG_ZOOKEEPER_CONNECT", "value": "zookeeper1:2181,zookeeper2:2181,zookeeper3:2181" },
        { "name": "ALLOW_PLAINTEXT_LISTENER", "value": "yes" }
      ]
    }
  ],
  "requiresCompatibilities": ["EC2"],
  "memory": "2048",
  "cpu": "1024"
}

💡 Tip: Use placementConstraints to spread tasks across Availability Zones for HA.

---

5. 💾 Enable Data Persistence

• ZooKeeper & Kafka require persistent storage

• Use EBS volumes mounted to ECS tasks (if EC2-backed)

• Or use Amazon EFS with Fargate

---

6. 🧪 Validate the Cluster

• Kafka CLI or client library can produce/consume:

    kafka-console-producer.sh --broker-list kafka1:9092 --topic test
    kafka-console-consumer.sh --bootstrap-server kafka1:9092 --topic test --from-beginning
  

• Use CloudWatch Logs kafka-topics.shand ZooKeeper CLI tools for health checks

---

🧯 High Availability Best Practices

---

📌 Conclusion

Apache Kafka and ZooKeeper bring critical capabilities to microservices: event streaming, durability, and fault-tolerant coordination. Deploying them on AWS ECS with high availability ensures

• Scalability across services

• Resilience to AZ or task failures

• Decoupled, event-driven communication

With ECS, you offload much of the orchestration while gaining flexibility to manage Kafka clusters as containerized services—without locking into managed services too early.