• ZooKeeper: A separate service (Apache ZooKeeper) used to manage Kafka metadata and cluster coordination.
• KRaft (Kafka Raft Metadata Mode): A newer, built-in Kafka consensus mechanism (introduced in Kafka 2.8, production-ready in Kafka 3.3+). Removes the ZooKeeper dependency by embedding metadata management inside Kafka itself.

---

🔹 How They Work

✅ ZooKeeper-based Kafka

• Kafka brokers rely on an external ZooKeeper cluster.
• ZooKeeper stores:
  • Broker registration
  • Topics, partitions, configuration metadata
  • Controller election (which broker is the active controller)
  • Access control lists (ACLs)
• When a broker joins/leaves, ZooKeeper notifies the controller.
• Controller then updates partition leadership and replication assignments.

---

✅ KRaft-based Kafka

• No ZooKeeper needed.
• Kafka brokers themselves run a Raft consensus algorithm (called KRaft).
• Metadata is stored in a special internal Kafka topic (__cluster_metadata) replicated across controller quorum nodes.
• One broker acts as the active controller, elected via Raft.
• Other brokers replicate metadata log, ensuring strong consistency.

---

🔹 Why the Change?

• ZooKeeper adds operational complexity → separate cluster to deploy & manage.
• Hard to scale → large clusters with thousands of partitions caused metadata bottlenecks.
• Inconsistencies could arise between ZooKeeper state and broker state.
• Kafka wanted a single distributed system, not dependent on an external one.

---

🔹 Advantages & Disadvantages

✅ ZooKeeper

Advantages:

• Battle-tested and stable (used for years).
• Clear separation between Kafka and coordination.

Disadvantages:

• Extra operational burden → need to maintain ZooKeeper cluster.
• Slower scaling when metadata grows.
• Communication overhead between ZooKeeper and brokers.
• Failures could cause metadata/controller inconsistencies.

---

✅ KRaft

Advantages:

• No external dependency → Kafka is self-contained.
• Better scalability → metadata stored in Kafka log and replicated efficiently.
• Faster recovery & controller election using Raft.
• Simplified deployment and management (fewer moving parts).
• Foundation for better exactly-once guarantees and future Kafka improvements.

Disadvantages:

• Newer → not as battle-tested as ZooKeeper (but rapidly maturing).
• Migration from ZooKeeper to KRaft requires planning and specific upgrade paths.
• Some older tooling/ecosystem still assumes ZooKeeper presence.

---

🔹 Summary Comparison

Feature	ZooKeeper Mode	KRaft Mode (Raft)
Metadata Storage	ZooKeeper nodes	Kafka internal topic __cluster_metadata
Controller Election	Done via ZooKeeper	Done via Raft quorum
Deployment Complexity	Requires external ZooKeeper cluster	Self-contained, no ZooKeeper
Scalability	Limited with large clusters	More scalable
Reliability	Stable, proven	Newer, but designed for stronger consistency
Future Support	Being phased out	Default going forward

---

🔹 Real-World Takeaway

• Old Kafka (pre-3.3) → ZooKeeper was required.
• Modern Kafka (3.3+) → KRaft mode is production-ready and recommended.
• Future → ZooKeeper support will be completely removed (likely around Kafka 4.x).