Apache Pulsar is a scalable messaging platform that relies on separate components: ZooKeeper for metadata management, BookKeeper for persistent storage, and the Pulsar Broker for messaging. This blog demonstrates setting up these components individually in Docker and initializing Pulsar metadata.

Prerequisites

Docker Installed: Ensure Docker is installed and running.
Pulsar Image: Use the official Apache Pulsar Docker image.
Network Configuration: Create a custom Docker network for communication.

Step 1: Setting Up ZooKeeper

ZooKeeper manages the metadata for Pulsar.

1.1 Start the ZooKeeper Container

Run the following command to deploy ZooKeeper in Docker:

docker run -d \
  --name zookeeper \
  -p 2181:2181 \
  apachepulsar/pulsar \
  bin/pulsar zookeeper

1.2 Verify ZooKeeper is Running

You can check the status of the ZooKeeper container:

docker ps

Step 2: Initialize Pulsar Metadata

Before starting BookKeeper and the Broker, you need to initialize metadata in ZooKeeper.

2.1 Initialize Metadata

Run the following command to initialize cluster metadata:

docker run -it --rm \
  --link zookeeper:zookeeper \
  apachepulsar/pulsar \
  bin/pulsar initialize-cluster-metadata \
  --cluster pulsar-cluster \
  --zookeeper zookeeper:2181 \
  --configuration-store zookeeper:2181 \
  --web-service-url http://localhost:8080 \
  --web-service-url-tls https://localhost:8443 \
  --broker-service-url pulsar://localhost:6650 \
  --broker-service-url-tls pulsar+ssl://localhost:6651

Step 3: Setting Up BookKeeper

BookKeeper handles message persistence for Pulsar.

3.1 Start the BookKeeper Container

Run the following command to deploy BookKeeper:

docker run -d \
  --name bookkeeper \
  --link zookeeper:zookeeper \
  -p 3181:3181 \
  apachepulsar/pulsar \
  bin/pulsar bookie

3.2 Verify BookKeeper

Ensure the container is running:

docker ps

Step 4: Setting Up Pulsar Broker

The Pulsar Broker handles client connections and message routing.

4.1 Create a Custom Network

Create a Docker network for communication:

docker network create pulsar-network

4.2 Connect Containers to the Network

Connect ZooKeeper and BookKeeper to the network:

docker network connect pulsar-network zookeeper
docker network connect pulsar-network bookkeeper

4.3 Start the Broker Container

Deploy the Pulsar Broker using the following command:

docker run -d \
  --name pulsar-broker \
  --network pulsar-network \
  -p 6650:6650 \
  -p 8080:8080 \
  apachepulsar/pulsar:latest \
  bin/pulsar broker \
  --zookeeper zookeeper:2181 \
  --bookkeeper-service-url bk://zookeeper:2181

4.4 Verify Broker Status

Check if the broker container is running:

docker ps

Step 5: Testing the Setup

5.1 Publish a Message

Use the Pulsar client to send a test message:

docker exec -it pulsar-broker \
  bin/pulsar-client produce persistent://public/default/test-topic \
  -m "Hello, Pulsar!"

5.2 Consume a Message

Receive the message using the Pulsar client:

docker exec -it pulsar-broker \
  bin/pulsar-client consume persistent://public/default/test-topic \
  -s "test-subscription" -n 1

Conclusion

This guide demonstrated how to set up Apache Pulsar components (ZooKeeper, BookKeeper, and Broker) separately in Docker. By deploying these components independently, you can scale and manage them efficiently. With Pulsar metadata initialized and the broker running, your Pulsar instance is now ready for production-grade messaging!

Setting Up Apache Pulsar in Docker: ZooKeeper, BookKeeper, and Broker