🔹 1. Kafka Data Replication
- Topic Partitions are replicated across brokers.
- Each partition has:
- Leader Replica → handles all reads/writes.
- Follower Replicas → copy data from the leader.
- Replication Factor (RF): number of replicas for a partition (e.g., RF=3 → 1 leader + 2 followers).
👉 If the leader fails, one of the in-sync followers (ISR) is promoted as the new leader.
🔹 2. Consumer Offsets Replication
- Consumer progress (offsets) is stored in an internal topic
__consumer_offsets, which is also replicated. - This ensures that even if a broker storing offsets fails, the offsets are safe.
- Keeps consumer group rebalancing consistent.
🔹 3. Transactional State Replication
- For exactly-once semantics (EOS), Kafka uses the
__transaction_stateinternal topic. - Stores transaction metadata and is replicated across brokers.
- Ensures committed/uncommitted messages can be recovered after broker failure.
🔹 4. Kafka Streams State Replication
- Kafka Streams applications store intermediate results in state stores (backed by RocksDB).
- These are checkpointed into changelog topics, which are replicated.
- If a stream task fails, another instance can rebuild state from the replicated changelog.
🔹 5. Connect Configs and Status Replication
- Kafka Connect uses internal topics:
__config→ connector configs.__status→ connector/task status.__offsets→ source offsets.
- All are replicated topics, so connector jobs can recover after failure.
🔹 6. Metadata Replication (ZooKeeper vs KRaft)
- In ZooKeeper mode: metadata (brokers, topics, ACLs, configs) stored in ZooKeeper, replicated across ZK quorum.
- In KRaft mode: metadata stored in an internal topic
__cluster_metadata, replicated using Raft protocol.
🔹 7. Log Segments (File System Level)
- On each broker, partitions are written as log segments on disk.
- These are not duplicated on the same broker, but replicated across brokers at the partition level.
🔹 8. Producer Reliability (Duplicates vs Idempotence)
- Retries: Producer may resend messages on failure → potential duplicates.
- Idempotent Producer (
enable.idempotence=true): ensures even with retries, duplicates are avoided. - Transactional Producer: duplicates avoided across multiple partitions/operations with EOS.
🔹 Summary Table
| Component | What is Replicated / Duplicated | Why (Failure Handling) |
|---|---|---|
| Partitions (data) | Replicated across brokers (RF ≥ 2) | Data durability & availability |
| Consumer Offsets | Internal topic __consumer_offsets (replicated) | Resume consumer progress after failure |
| Transaction State | Internal topic __transaction_state (replicated) | Ensure exactly-once delivery |
| Streams State Stores | Changelog topics (replicated) | Rebuild state after task crash |
| Connect Configs/Offsets | Internal topics __config, __status, __offsets | Connector/task recovery |
| Cluster Metadata | ZooKeeper quorum or __cluster_metadata topic (KRaft) | Broker/topic/ACL/config recovery |
| Producer Messages | Retries (duplicates possible), Idempotence removes dups | Ensure reliability |
✅ In short: Kafka replicates partitions, metadata, offsets, transactions, and internal system topics to make sure failures (broker crash, consumer crash, or connector failure) do not lose data or state.