From ActiveMQ to Kafka: My Journey to Understanding Why We Still Need Queues (KIP-932)

Nine years ago, my amazing former boss and mentor Ajith Kumar, asked me to do an impact analysis on replacing ActiveMQ with Kafka in our integration layer. It was part of a broader product overhaul. We took the plunge and adopted Kafka across the stack, replacing ActiveMQ and IBM Infosphere Streams.

I was thrilled.

Suddenly, no more duplicating messages for multiple consumers. No more clunky back-pressure issues. Fast, efficient fan-out. Cleanly decoupled microservices. Kafka felt like a breath of fresh air — a single, elegant solution to many of our messaging woes.

For long afterwards, I’d sip my coffee and smugly think, “Why do people even talk about queues anymore?” Kafka could send messages, handle multiple consumers, support retries... What more could you need?

As it turns out, quite a bit. And not in a bad way — just in a nuanced way. Let’s unpack it - without getting too technical about it.

---

Kafka: The Mighty Swiss Army Knife (Missing a Few Attachments)

Kafka is incredibly powerful. It moves data at scale, connects systems in real time, and keeps microservices humming along. It’s the de-facto tool for anything to do with real time data and modern event-driven architectures.

But even the best Swiss Army knife isn’t the ideal tool for every job. If you’re opening a wine bottle, a dedicated corkscrew still wins — cleaner, simpler, less likely to injure your hand.

The same is true for messaging models.

---

Two Models, Two Mindsets

1. Publish-Subscribe (Pub-Sub): The Loudspeaker

Imagine you're at a party and someone yells, “FREE PIZZA IN THE KITCHEN!” Everyone who wants pizza hears it. Those who aren’t interested - or are already three beers deep 🍻🍻 - move on.

Pros:

• One-to-Many: Say it once, deliver to all interested consumers.

• Decoupling: The speaker doesn’t care who hears it. Listeners come and go.

• Scalability: Producers and consumers scale independently.

Cons:

• Task Ownership? If someone yells “CLEAN THE KITCHEN,” who actually does it?

• No Built-in Progress Tracking: There’s no concept of one message = one worker = one result.

2. Queues: The To-Do List

Now imagine a whiteboard with "Clean Kitchen" written on it. The first person to see it claims it, does the job, and crosses it off.

Pros:

• Clear Ownership: One message, one worker.

• Workflow Friendly: Easy to track state — pending, in progress, done.

• Built-in Retries: If someone can’t finish a task, it goes back in the queue.

Cons:

• No Broadcasts: Not ideal for notifying multiple parties at once.

• Potential Bottlenecks: Tasks can pile up if processing slows down.

---

An epiphany in the Queue

The moment when I truly started appreciating the pub-sub and queue semantics.

Last year, I was helping a telco client build observability and monitoring layer around their Confluent Kafka setup. Out of curiosity, I asked how they used Kafka (in a professional and monitoring way 🤓).

They walked me through their order processing pipeline. Orders flowed through several stages: Validation → Provisioning → Billing → Activation, and so on. Each stage had a corresponding Kafka topic.

If an order passed validation, it got published to the Provisioning topic. Critically, an order only moves to the next stage if everything checks out in the current stage. Simple enough.

But if validation failed due to a transient issue (say, a temporary credit check error), they’d re-publish the same message back to the Validation topic for it to be retried again later.

Me: “How long do you retry?”
Them: “We increment a counter as a part of the Kafka message. If it crosses a threshold, we stop and send it to a dead-letter topic.”

Here’s the rub: Kafka pub-sub was doing queue work - poorly. There was no natural way to handle retries, no ownership semantics, no stateful tracking. Just a lot of manual handling and topic juggling.

It worked — but awkwardly. Like opening a bottle of wine with a butter knife.

---

Enter Queue Semantics in Kafka (KIP-932)

Now imagine if they could simply say:

• ACCEPT – “Validation succeeded. Move it along.”

• RELEASE – “Temporary hiccup. Retry later.”

• REJECT – “This order is bad. Send it to manual review.”

With proper queue semantics (thanks to KIP-932), this is now possible in Kafka. No more simulating queues with topics and retries. The system itself can track message states, retries, and dead letters — without the consumer doing cartwheels.

Even better? You can mix pub-sub and queue semantics on the same topic. Need fan-out? Sure. Need one-worker-per-message guarantees? Also yes.

---

The Takeaway: It's Not Either/Or — It’s Both

My old belief — “Kafka can do everything!” — wasn’t wrong. But it was simplistic. Kafka can simulate queues, yes. But now, with native queue semantics, it doesn’t have to.

Use pub-sub when you want to shout about pizza.

Use queues when someone needs to clean the kitchen.

And now, with modern Kafka, you don’t have to choose between one model or the other — you just have to pick the right semantic for the job.

Versatility and power - that is what KIP-932 brings to Kafka.