Keep data safe with replication

The short version

Each partition is copied to several brokers. One copy is the leader and takes all writes. The others are followers that copy the leader. The replication factor is the number of copies.

Details

Followers that are fully caught up are the in sync replicas, shown as Isr. With acks set to all, a write succeeds only when every in sync replica has it. If the leader dies, Kafka promotes a follower from that list. Production clusters usually use a replication factor of 3.

Try it yourself

Ask for three copies on a cluster with one broker:

docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --create --topic payments --replication-factor 3

You should see this error:

Error while executing topic command : Replication factor: 3 larger than available brokers: 1.

Now describe orders. Every partition lists the same single broker as its leader, its only replica and its only in sync replica.

Explain it to the duck

Why does Kafka refuse to create more copies than there are brokers?

Tested with: Kafka 4.1.0 in Docker.