Kafka Producer Configuration

Kafka Producer Overview

Writing messages to Kafka serves many purposes: enabling asynchronous communication between microservices, buffering data before database writes, event sourcing, and more. Each use case has unique demands:

• Is every message critical, or can some be lost?

• Can your system tolerate duplicate messages?

• Are you optimizing for latency or throughput?

These questions shape how you configure the Kafka producer for your needs.

Kafka Producer Configuration

---

1. bootstrap.servers (Mandatory)

What: List of Kafka brokers to initially connect to.
Why: Producer must know where Kafka cluster lives.
How: Comma-separated list, e.g., "broker1:9092,broker2:9092"
When: Always set explicitly. No default.

---

2. key.serializer and value.serializer (Mandatory)

What: Convert keys/values to bytes.
Why: Kafka protocol is binary; these serializers transform objects to byte arrays.
How: Use built-in (e.g., StringSerializer) or implement custom serializers.
When: Always set according to your data types.

---

3. acks

What: Number of acknowledgments broker must receive before considering a send successful.

• 0: No ack (fire and forget, lowest latency, possible data loss)

• 1: Leader ack (default, balances latency and durability)

• all (or -1): Wait for all in-sync replicas (strongest durability)
  Why: Controls durability guarantees.
  When:

• Use all if every message is critical.

• Use 0 if ultra-low latency and some loss is acceptable.
  Nuance: Higher acks increase latency but improve durability.

---

4. buffer.memory

What: Total memory (bytes) available to buffer records waiting to be sent.
Why: Controls how much data the producer can hold before blocking or dropping.
How: Set size in bytes (default ~32MB).
When: Increase for high throughput workloads.
Nuance: Too small causes frequent blocking.

---

5. compression.type

What: Compression codec used on batches (none, gzip, snappy, lz4, zstd).
Why: Reduces network bandwidth and disk usage.
When: Use compression in high-throughput environments or where network is a bottleneck.
Nuance: Compression adds CPU overhead, choose codec based on CPU/network tradeoff.

---

6. retries

What: Number of retry attempts for failed sends.
Why: Improves reliability under transient broker/network failures.
How: Integer, default 0.
When: Increase to improve reliability, but beware of message duplication without idempotence.
Nuance: Combine with max.in.flight.requests.per.connection=1 to avoid out-of-order retries.

---

7. batch.size

What: Maximum size (bytes) of a batch of records per partition.
Why: Larger batches improve throughput but increase latency.
How: Default 16KB; increase for high throughput.
When: Increase for bulk workloads; keep low for low latency.

---

8. linger.ms

What: How long producer waits before sending a batch, allowing more records to accumulate.
Why: Controls tradeoff between latency and throughput.
How: Default 0 (send immediately).
When: Increase (e.g., 5-50ms) to improve throughput with small latency cost.

---

9. client.id

What: Identifier for the producer client.
Why: Useful for logging, monitoring, and quota enforcement.
How: Any string.
When: Always set in multi-producer environments.

---

10. max.in.flight.requests.per.connection

What: Max number of unacknowledged requests per connection.
Why: Controls potential for message reordering on retries.
How: Default 5.
When: Set to 1 for strict ordering (with retries).
Nuance: Higher value = better throughput, but can reorder messages on retry.

---

11. Timeouts

• timeout.ms: Deprecated, use below.

• request.timeout.ms: Max time to wait for a response before failing a request (default 30s).

• metadata.fetch.timeout.ms: Timeout for fetching metadata from broker (default 60s).
  Why: Controls responsiveness and failure detection.
  When: Tune in slow or unreliable networks.

---

12. max.block.ms

What: Max time producer blocks during buffer exhaustion or metadata fetching.
Why: Controls how long send() or partitionsFor() calls block.
How: Default 60s.
When: Lower to fail fast in high availability apps.

---

13. max.request.size

What: Max size of a request sent to the broker.
Why: Prevents sending huge messages that can overwhelm brokers.
Default: 1MB.
When: Increase if your messages are large.

---

14. Network Buffers

• receive.buffer.bytes and send.buffer.bytes control OS socket buffer sizes.

• Defaults depend on OS, usually 64KB or 128KB.
  Why: Tuning can improve throughput and latency in high traffic environments.

---

Example: Basic Producer Configuration in Spring Boot (Java)

import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;

Properties props = new Properties();

// Mandatory: List of Kafka brokers to connect
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");

// Mandatory: Serializers convert keys and values to bytes for network transmission
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());



// ---------- Durability & Reliability ----------


// Default: "1" (leader ack only)
// acks controls message durability:
// "all" waits for all ISR replicas to confirm for max durability.
// Use "all" if losing any message is unacceptable (e.g. payment systems).
// Use "1" or "0" for lower latency but risk message loss.
props.put(ProducerConfig.ACKS_CONFIG, "all");

// Default: 0
// Number of retry attempts on transient errors.
// Increase retries if temporary network/broker glitches occur frequently.
// Too many retries may delay failure detection.
props.put(ProducerConfig.RETRIES_CONFIG, 5);

// Default: 5
// Controls how many unacknowledged requests can be sent concurrently per connection.
// Setting to 1 preserves message order on retries (important for ordered topics).
// Higher values increase throughput but may cause message reordering on retry.
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);




// ---------- Performance Tuning ----------


// Default: 16 KB (16384)
// Higher = better throughput, smaller = lower latency
// Batch size (in bytes) controls how many bytes of records to collect before sending.
// Larger batch sizes improve throughput by amortizing overhead.
// Set small batches if low latency is more important than throughput.
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 32768); // 32 KB



// Default: 0 ms
// Time to wait for more records to fill batch
// Trade-off between latency and throughput
// linger.ms adds artificial delay to wait for more records before sending batch.
// Higher linger can improve throughput by increasing batch fullness.
// Use low linger for low latency scenarios.
props.put(ProducerConfig.LINGER_MS_CONFIG, 10); // Wait max 10ms before sending batch


// Default: "none"
// Compress messages to save network bandwidth (gzip, lz4, snappy, zstd)
// May use more CPU; "snappy" is a good balance
// Compression reduces network usage and storage space.
// Supported: none, gzip, snappy, lz4, zstd.
// Use compression to save bandwidth and disk but beware of CPU overhead.
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");






// ---------- Memory and Blocking ----------


// Default: 32 MB (33554432)
// Max memory for buffering unsent records
// Tune if your producer sends bursts or large batches
// buffer.memory sets max bytes the producer can buffer records in memory before sending.
// Increase if producing bursts of messages to avoid blocking.
// If buffer full, producer blocks or throws exception after max.block.ms.
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 67108864L); // 64 MB


// Default: 60,000 ms
// Max time to block if buffer is full or metadata unavailable
// Prevents app from hanging indefinitely
// max.block.ms is max time producer blocks on buffer full or metadata fetch.
// If exceeded, producer throws TimeoutException.
// Tune according to how long your application can tolerate blocking.
props.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, 60000); // 60 seconds





// ---------- Request Limits ----------


// Default: 1 MB (1048576)
// Max total request size (all records + overhead)
// Increase if sending large messages or batches
// Request size limits max size per request including all records.
// Prevents sending oversized requests that brokers reject.
// Increase if sending large records or batches.
props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 1048576); // 1 MB default

// Default: OS-dependent (~64 KB or 128 KB typical)
// Network I/O buffers. Increase for high throughput, leave default unless needed
// Socket buffer sizes for network I/O.
// Larger buffers can improve throughput on high-latency or high-bandwidth networks.
// Defaults usually suffice; tune only if network performance issues.
props.put(ProducerConfig.RECEIVE_BUFFER_CONFIG, 65536); // 64 KB
props.put(ProducerConfig.SEND_BUFFER_CONFIG, 131072);   // 128 KB





// ---------- Monitoring ----------

// Default: "" (empty string)
// Logical identifier for this producer. Useful in logs and metrics
// client.id identifies producer in broker logs and metrics.
// Useful for monitoring multiple producers.
props.put(ProducerConfig.CLIENT_ID_CONFIG, "my-producer");

---

Summary

Kafka producer configs let you fine-tune:

• Durability (acks, retries)

• Ordering (max.in.flight.requests.per.connection)

• Performance (batch.size, linger.ms, compression.type)

• Resource use (buffer.memory, network buffers)

• Timeouts and error handling

Choose defaults for simple use cases; customize for critical, large-scale, or low-latency systems. Understanding each config’s role helps you build producers tuned for your unique workload.