Kafka: Replications and ISR

In Apache Kafka, the replication factor refers to the number of copies (replicas) of a topic partition that are maintained across the Kafka cluster.

ISR stands for In-Sync Replicas. It is the set of all replicas (including the leader replica) that are in sync with the leader of a Kafka partition.

For example, if the replication factor is set to 3, Kafka will maintain 3 copies of the data for that partition across different brokers. (1 leader + 2 followers)

One replica is designated as the leader. All read and write operations for the partition are handled by this leader. Other replicas are followers, which replicate data from the leader to stay in sync.

Importance of ISR

High Availability: If the leader replica fails, one of the replicas in the ISR can take over as the new leader, ensuring no data loss or downtime.

Durability Guarantees: Kafka can be configured to acknowledge writes only when data is written to all replicas in the ISR (acks=all), which ensures stronger durability.

Replication Lag Monitoring: Kafka continuously monitors replicas to ensure they remain in sync. If a replica falls behind for too long, it is removed from the ISR.

Trade-Offs

Increased Fault Tolerance vs. Resource Usage: Each replica consumes additional disk space on the brokers. For example, a replication factor of 3 triples the storage requirement for the same data.

High Availability vs. Performance Overheads: A higher replication factor increases the time to propagate writes across all replicas. This is especially significant when using acks=all, as the leader waits for acknowledgments from all in-sync replicas (ISR) before confirming the write.

---

Hope this helps to get some understanding of How Replication work in Kafka. Let me know if you have any other questions.

Resources

• kafka.js.org

• kafka.apache.org