Event Driven Architecture: Kafka and RabbitMQ
Comparing Kafka and RabbitMQ for an event-driven architecture, and why RabbitMQ won for a 500–1000 TPS workload
Background
Software often starts small, handling every request the same monotonous way. The architecture only gets revisited when new information surfaces (e.g. an expected RPS/TPS target) or an incident forces the issue (e.g. throttling under load).
On one project, the product team expected the system — freshly approved after its PoC — to withstand 500–1000 TPS (Transactions Per Second) at peak. That gave the software team a reason to step back and re-design the architecture around an EDA (Event Driven Architecture).
A message broker helps a throughput ceiling in two ways: it buffers bursts so a spike doesn’t slam the database or a downstream service all at once, and it decouples producers from consumers so a slow consumer can’t back-pressure the request path. The write returns as soon as the event is enqueued; the heavy processing happens asynchronously.
Summary
The two final contenders were Kafka and RabbitMQ. The key differences:
| Kafka | RabbitMQ |
|---|---|
| Messages live in it | Messages flow through it |
| Dumb broker / smart consumer | Smart broker / dumb consumer |
| Self-paced (pull-based) | Low latency (push-based) |
| Extremely high throughput (millions of events/sec) | Moderate throughput (thousands of messages/sec) |
Weighing the product team’s expected write traffic against the cloud infrastructure budget, RabbitMQ was the final choice. Thousands of messages per second comfortably covers 500–1000 TPS, and Kafka’s headline strengths — million-event throughput, long retention, and replay — weren’t requirements here. Paying for that headroom, in both infrastructure and operational complexity, made little sense.
The main thing we gave up by not choosing Kafka is replay: Kafka consumers track an offset and can re-read history, whereas RabbitMQ deletes a message on acknowledgement. We had no audit or replay requirement, so it was an acceptable trade.
The rest of this post walks the producer and consumer mechanics behind these tradeoffs.
Kafka
Well known for:
- Extremely high throughput
- Replay
- Fan-out
- Data retention
Overall Flow
Kafka: producers and consumers connect to the broker’s partitions over persistent TCP
Producer Flow
Producer pushes an event to a partition, gets an ACK back, then confirms the write to its caller
Consumer Flow
The consumer asks the broker for events to process. Each event lives in the broker’s partition until its retention date — the consumer pulls at its own pace.
RabbitMQ
Well known for:
- Complex message routing
- Messages intended for a single consumer
- Moderate data volumes
Overall Flow
RabbitMQ: producers publish to exchanges, which route to queues that consumers drain
Producer Flow
The exchange enables the complex routing Kafka does not support: the producer pushes an event, the exchange routes it to a queue, then ACKs
Consumer Flow
The broker pushes a message to a consumer. Once the ACK returns, the message is deleted from the queue.
Because the ACK deletes the message, it is a commitment. If a consumer crashes before ACKing, RabbitMQ requeues the message for another consumer — at-least-once delivery, meaning the same event can arrive twice, so consumers must be idempotent. Messages that keep failing get routed to a dead-letter queue rather than looping forever.
Choosing Between Them
The project’s decision was specific to its traffic and budget; the general rule is broader:
- Reach for Kafka when you need replay, long retention, high fan-out, or ordered streams at massive scale — event sourcing, log pipelines, analytics.
- Reach for RabbitMQ when you need flexible routing, per-message acknowledgement, and moderate volume at low latency — task queues, RPC, workflow orchestration.
In Action
Live demo: RabbitMQ in action