Build an order pipeline

The short version

Build a small pipeline. A producer sends orders keyed by customer, a processor turns them into kitchen tickets, and a group of two workers handles the tickets.

Details

Build it in this order, and check each part works before you add the next.

1. Create orders and tickets with 3 partitions each.
2. Write a producer that sends 20 orders for 4 customers, with the customer as the key and acks set to all.
3. Write a processor that reads orders, writes one ticket for each order and commits after each write.
4. Run two workers in the group kitchen that read tickets, print each ticket and commit after printing.
5. Stop one worker halfway through, then start it again.

Try it yourself

You are done when all of these are true:

• Every order is printed as a ticket, and none are lost when a worker stops.
• Tickets for one customer print in the order they were sent.
• Describing the kitchen group shows a lag of 0 on every partition.
• A short README explains why you chose your key and when you commit.

Explain it to the duck

Walk through what happens to one order if the processor crashes right after writing the ticket but before committing.

Tested with: Kafka 4.1.0 in Docker, Python 3.12 and confluent-kafka 2.11.