4. Starting a Apache Pulsar Service in Microk8s

Introduction to Apache Pulsar

Apache Pulsar is a distributed messaging and streaming platform designed for high-performance, multi-tenant applications. It supports pub-sub messaging, event processing, and real-time analytics. Pulsar is ideal for applications that require scalability, durability, and low-latency message delivery.

In this guide, we will go through a detailed step-by-step process to install and configure Apache Pulsar in a MicroK8s environment. This setup ensures that you can efficiently run Pulsar within a lightweight Kubernetes distribution.

---

1. Add and Update Helm Repository

First, add the Apache Pulsar Helm repository to your Helm installation:

helm repo add apache https://pulsar.apache.org/charts

If the repository already exists, Helm will skip adding it. Next, update your local Helm chart repository to fetch the latest Pulsar charts:

helm repo update

---

2. Configure Pulsar Components

Before deploying Pulsar, define the components you need by creating a configuration file. Below is an example configuration enabling essential components (do not use this for your installation):

# Enable Pulsar components
zookeeper: true
bookkeeper: true
autorecovery: true
broker: true
functions: true
proxy: true
toolset: true
pulsar_manager: true

Zookeeper and BookKeeper are essential for metadata management and message durability. The broker handles client requests, while the proxy simplifies access to Pulsar clusters.

---

3. Create Custom Configuration

To customize Pulsar deployment, create a custom.yaml file with the following minimal configuration:

zookeeper:
  replicaCount: 1
bookkeeper:
  replicaCount: 1
broker:
  replicaCount: 1
  configData:
    managedLedgerDefaultEnsembleSize: "1"
    managedLedgerDefaultWriteQuorum: "1"
    managedLedgerDefaultAckQuorum: "1"
proxy:
  replicaCount: 1

This configuration ensures a lightweight Pulsar deployment with minimal resources for testing and development. But do note that the number of pods for each of the service is limited to one and hence there would be no failover mechanism applicable to this.

---

4. Deploy Pulsar

Before deployment, verify the template:

helm template pulsar apache/pulsar

Generate the template using the custom values:

helm template --values=custom.yaml pulsar apache/pulsar

Now, install Pulsar in the pulsar namespace:

helm install --values custom.yaml --set initialize=true --namespace pulsar pulsar apache/pulsar

Upon successful installation, you should see:

NAME: pulsar
LAST DEPLOYED: <timestamp>
NAMESPACE: pulsar
STATUS: deployed
REVISION: 1

---

5. Verify Installation

To confirm that Pulsar is running correctly, check the Helm release:

helm ls -n pulsar

Ensure that all Pulsar components are running. This should be the first way to check if you pulsar installation is ready or not. Typically the below command should list available pods and should also mention the state of the pods:

kubectl get pods -n pulsar

To verify the created services and to understand the service type and port mappings that each service is using, run the below command:

kubectl get svc -n pulsar

---

6. Access and Verify Broker

To interact with the broker (to enter into the broker container), execute the following command:

kubectl exec -it pulsar-broker-0 -n eventing -- /bin/bash

Verify the Pulsar cluster:

./bin/pulsar-admin clusters list

Expected output:

pulsar

---

7. Check Client Configuration

To inspect the Pulsar client configuration, access the toolset pod:

kubectl exec -it -n eventing pulsar-toolset-0 -- /bin/bash

View the configuration file:

cat conf/client.conf

Pulsar REST API URL:

webServiceUrl=http://pulsar-proxy:80/

Binary Protocol URL for publishing and subscribing:

brokerServiceUrl=pulsar://pulsar-proxy:6650

---

8. Access Grafana Dashboard

To retrieve Grafana's admin password:

kubectl get secrets pulsar-grafana-secret -n pulsar -o jsonpath='{.data.GRAFANA_ADMIN_PASSWORD}' | base64 --decode

Use these credentials to log in to the Grafana dashboard. If the above command does not work then it is better to get the entire yaml file that contains the credentials and then take the credentials from the yaml file and apply base64 decode to use them.

Grafana can be accessed via:

http://YOUR_STATIC_IP:32001

---

9. Setting Up Grafana in MicroK8s

Step 1: Delete Existing Grafana Service

kubectl delete svc pulsar-grafana -n pulsar

Step 2: Apply New NodePort Service Configuration

Create a service file pulsar-grafana-service.yaml and apply it:

kubectl apply -f pulsar-grafana-service.yaml -n pulsar

Verify the service:

kubectl get svc -n pulsar

Step 3: Retrieve Grafana Credentials

Get the stored secret:

kubectl get secret pulsar-grafana -n pulsar -o yaml

Decode the base64 credentials:

echo YWRtaW4= | base64 --decode  # Decode username
echo bmdPT0lwRjVIT0oyTE9MREJrMlhnMUgwN25pM09kSk5HbVE3d1FEYw== | base64 --decode  # Decode password

---

Conclusion

By following this guide, you have successfully installed and configured Apache Pulsar on MicroK8s. You also set up monitoring using Grafana. This setup is ideal for testing, development, and small-scale production environments.

For larger deployments, consider scaling Pulsar components and fine-tuning configurations based on workload requirements.