Kafka Producer Architecture

Kafka producers are responsible for sending data to Kafka topics. Let’s break down how a Kafka producer works internally, step by step, based only on what is required to deliver a message reliably and efficiently.

---

Kafka Producer Components

Producer Record This is the data unit created by the application, containing:

1. • The topic to which the record should be sent

     • The value (the actual data)

     • Optionally a key and partition

2. Serialization Kafka operates over the network using bytes.

   • The key and value objects are serialized into byte[]

   • Kafka provides built-in serializers and allows custom implementations

3. Partitioning

   • If a partition is specified in the ProducerRecord, Kafka uses it directly

   • If not, Kafka uses the partitioner component to select a partition

     • This is usually done by hashing the key

     • No key? Kafka uses a round-robin strategy

4. Batching Once a partition is chosen:

   • The producer knows the exact topic and partition

   • The serialized record is added to a batch (a memory buffer for the same topic-partition pair)

5. Sender Thread

   • A separate background thread picks up the batches

   • It sends the batch to the appropriate Kafka broker

6. Broker Response

   • Upon receiving the record, the broker returns a response

   • If successful, a RecordMetadata object is returned containing:

     • Topic

     • Partition

     • Offset

   • If unsuccessful, an error is returned

     • The producer may retry a few times before giving up and surfacing the error

---

Summary :

First, a ProducerRecord is created and sent using the Producer,after which the serializer serializes the key and value into bytes, then passes them to the Partitioner to determine the partition if no partition is set, after which the record is added to a batch for the resolved topic-partition, and finally a background sender thread transmits the batch to the appropriate Kafka broker, which responds with either RecordMetadata on success or an error that may trigger a retry.

They are sent in batches of multiple records for the same topic-partition to optimize network usage and improve throughput by sending them together in fewer requests.