Share work with a consumer group

The short version

Consumers that share a group.id split the partitions of a topic between them. Each partition goes to exactly one consumer in the group, so the group handles every message once.

Details

A second group gets its own copy of every message. That is the difference from a classic queue: one group behaves like a queue, and many groups behave like publish and subscribe. A group can use at most as many consumers as there are partitions. Extra consumers wait idle until one leaves.

Try it yourself

Run this in two terminals:

docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic orders --group kitchen

In a third terminal, describe the group:

docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --describe --group kitchen

You should see three rows, one for each partition, with two different values in the CONSUMER-ID column.

Explain it to the duck

Your topic has 3 partitions and you start 5 consumers in one group. What do the extra two do, and why?

Tested with: Kafka 4.1.0 in Docker.