Apache-kafka with raft mode in Docker Compose

Introduction to Raft Mode:

• Raft mode in Kafka refers to a new architecture introduced to remove ZooKeeper from Kafka’s setup.

• ZooKeeper used to manage Kafka’s metadata (like topic configurations and broker states). However, ZooKeeper adds complexity.

• With Raft mode (also called KRaft mode), Kafka now handles its own metadata using an algorithm called Raft.

• In Raft, there’s a leader and multiple followers. The leader takes decisions, and the followers replicate everything the leader does to maintain consistency.

Setting Up Kafka with Raft Mode Using Docker Compose:

version: '3.8'

services:
  kafka:
    image: confluentinc/cp-kafka:7.2.1
    #container settings
    container_name: kafka-kraft
    ports:
      - "9092:9092"    # Port for client connections (producer/consumer)
      - "9093:9093"    # Port for internal controller communication
    #Environment variables
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@localhost:9093'
      KAFKA_PROCESS_ROLES: 'broker,controller'

This Docker Compose file sets up Kafka in KRaft mode, which means Kafka is running without ZooKeeper and using the Raft consensus algorithm for managing metadata.

• version: '3.8': Specifies the version of Docker Compose being used.

• services: Defines the services you want to run. In this case, the only service is Kafka.

  Container Settings:

• services: Defines the services you want to run, in this case, Kafka.

• kafka: This is the service name for Kafka.

• image: confluentinc/cp-kafka:7.2.1: It uses the cp-kafka image from Confluent’s repository, version 7.2.1, which includes Kafka with support for KRaft mode.

• container_name: kafka-kraft: Names the container kafka-kraft, making it easier to refer to it within Docker.

• ports:

  • "9092:9092": Exposes port 9092 for Kafka’s PLAINTEXT listener used by clients (producers and consumers).

  • "9093:9093": Exposes port 9093 for controller communication between Kafka brokers in KRaft mode.

Environment Variables:

• KAFKA_NODE_ID: 1: Assigns a unique ID to the Kafka broker, which is important for identifying brokers in the cluster.

• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: Defines how different listeners (e.g., PLAINTEXT, CONTROLLER) will handle security protocols.

  • PLAINTEXT: Unencrypted communication for producers and consumers.

  • CONTROLLER: Unencrypted communication between brokers in KRaft mode.

• KAFKA_LISTENERS: Specifies the ports and protocols Kafka will listen on.

  • PLAINTEXT://localhost:9092: For regular client communication (producers/consumers).

  • CONTROLLER://localhost:9093: For internal controller communication.

• KAFKA_ADVERTISED_LISTENERS: Tells Kafka which listener address to advertise to clients. Here, it advertises the PLAINTEXT listener on port 9092.

• KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER': Defines which listener Kafka will use for internal controller communication. In this case, it's the CONTROLLER listener on port 9093.

• KAFKA_CONTROLLER_QUORUM_VOTERS: '1@localhost:9093': Specifies the brokers participating in the controller quorum (i.e., the decision-making process in KRaft mode). Since this is a single-node setup, only one broker (1) is specified, with its communication on port 9093.

• KAFKA_PROCESS_ROLES: 'broker,controller': This specifies that this broker will act both as a Kafka broker (handling client requests) and as a controller (managing cluster metadata) in KRaft mode.

Problem: Running Kafka with Raft Mode on Local Machine

1. When using the Docker Compose file above, you may encounter the error.

   Even though Raft mode doesn’t require ZooKeeper, the Confluent Kafka image might still expect it due to default settings.

    KAFKA_ZOOKEEPER_CONNECT is required.
   

#!/bin/sh

# Remove the requirement for KAFKA_ZOOKEEPER_CONNECT
sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure

# Prevent the startup script from checking for ZooKeeper readiness
sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure

# Format the Kafka storage if needed
echo "kafka-storage format --ignore-formatted -t <uniqueId - replace> -c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure

• Put this in the same folder, where you put docker-compose file.

• On Windows: If you’re using Windows, use Git Bash to modify file permissions:

•   chmod +x kraft-fix.sh
  

• If you used Notepad++ to create the file, make sure to convert it to UNIX format (Edit → EOL Conversion → UNIX).

• 

  Update the Docker Compose File:

  Next, we need to modify the Docker Compose file to mount and execute the kraftfix.sh script inside the container.

        volumes:
          - ./kraftfix.sh:/tmp/kraftfix.sh
        command: "bash -c '/tmp/kraftfix.sh && /etc/confluent/docker/run'"
  

  • volumes: Mounts the kraftfix.sh file inside the Docker container.

  • command: Runs the script before starting Kafka.

• Run the Docker Compose File:

  Once everything is set up, start Kafka using the following command:

•   docker-compose up
  

Conclusion:

By setting up Kafka with Raft mode:

• No ZooKeeper: Kafka can now manage everything itself, reducing external dependencies.

• Simpler Setup: Less complexity in managing multiple components.

• Better Scalability: Raft makes Kafka easier to scale and maintain.

Raft ensures that Kafka clusters remain fault-tolerant and consistent even when some brokers fail, without needing ZooKeeper.