Watch consumer lag

Operating a cluster · Article 10 of 12 · 3 min read

Lag is the first number to check when a consumer falls behind or stops.

The short version

Lag is the number of messages between the committed offset of a group and the newest message in a partition. Rising lag means consumers are slower than producers.

Details

kafka-consumer-groups.sh --describe shows CURRENT-OFFSET, LOG-END-OFFSET and LAG for each partition. Lag that grows steadily needs more consumers or faster handling. Lag stuck on one partition often points to one bad message or one stuck consumer. In production, export lag as a metric and alert on it.

Try it yourself

Stop every kitchen consumer. Run produce.py five times, then describe the group:

docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --describe --group kitchen

You should see the LAG column add up to 5. Start a kitchen consumer, describe the group again, and lag is back to 0.

Explain it to the duck

Lag is growing on one partition only. What would you check first, and why?

Tested with: Kafka 4.1.0 in Docker.