Read messages from code

The short version

Create a consumer, subscribe to a topic, then call poll in a loop. Each call returns one message, or nothing if none arrived in time.

Details

Every consumer needs a group.id. You will see why in the next article. Setting auto.offset.reset to earliest makes a new group start from the oldest message instead of waiting for new ones. Always close the consumer so Kafka knows it has left.

from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "printer",
    "auto.offset.reset": "earliest",
})
consumer.subscribe(["orders"])

try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            print(msg.error())
            continue
        print(msg.key().decode(), msg.value().decode(), msg.offset())
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

Try it yourself

Save the code as consume.py, run it, wait a few seconds and press Ctrl+C. Then run it again.

You should see one line for each order you sent, such as alice pizza 0. The second run prints nothing.

Explain it to the duck

Why does the second run print nothing, even though Kafka still has every message?

Tested with: Kafka 4.1.0 in Docker, Python 3.12 and confluent-kafka 2.11.