Send messages from code

The short version

Create a producer, call produce, then call flush to wait until Kafka confirms. Set acks to all so a write only counts once every copy of the partition has it.

Details

produce only queues the message. Kafka confirms it later through a delivery callback. flush waits until every queued message is confirmed or has failed. With idempotence turned on, a retry never writes the same message twice.

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "localhost:9092",
    "acks": "all",
    "enable.idempotence": True,
})

def report(err, msg):
    print(err or f"partition {msg.partition()} offset {msg.offset()}")

producer.produce("orders", key="alice", value="soup", on_delivery=report)
producer.flush()

Try it yourself

Install the client with pip install confluent-kafka, save the code as produce.py and run python produce.py twice.

You should see a line such as partition 1 offset 2 each time. The offset goes up by one, and the partition stays the same because the key is the same.

Explain it to the duck

What is the difference between calling produce and the message being safely stored?

Tested with: Kafka 4.1.0 in Docker, Python 3.12 and confluent-kafka 2.11.