Learn when a queue makes your system faster and more resilient, how retries, idempotency and dead-letter queues work, and which broker fits your needs.
Imagine a customer places an order. Your server charges the card, reserves stock, sends a confirmation email and updates analytics, all before the customer sees a response. If the email service is slow, the customer waits. If it is down, the order fails. A message queue removes that coupling by letting the slow or unreliable parts happen after the response.
Synchronous versus asynchronous
In a synchronous design, service A calls service B and waits. A is only as fast and as reliable as B.
In an asynchronous design, A puts a message on a queue and carries on. B picks the message up when it can. If B is down, the message waits. If B is slow, the queue absorbs the spike.
That is the whole idea. The rest of this article is about doing it safely.
A queue in action
sequenceDiagram
participant O as Order service
participant Q as Message queue
participant P as Payment service
participant I as Inventory service
participant E as Email service
participant DLQ as Dead-letter queue
O->>Q: Publish order event
Q->>P: Process payment
P-->>Q: Payment confirmed
Q->>I: Reserve inventory
I-->>Q: Inventory reserved
Q->>E: Send confirmation
E-->>Q: Email sent
Q-->>O: Order processed
alt Payment failure
P-->>Q: Payment failed
Q->>DLQ: Move failed event
else Inventory failure
I-->>Q: Reservation failed
Q->>DLQ: Move failed event
else Email failure
E-->>Q: Delivery failed
Q->>DLQ: Move failed event
end
The order service answers the customer as soon as the order is saved and the event is published. The payment, inventory and email work happens independently. Anything that fails repeatedly lands in a dead-letter queue, where it waits for a person or a repair job.
Messages arrive at least once, so handlers must be idempotent.
Queue or publish/subscribe
Two patterns are easy to confuse:
- Work queue. Each message is handled by one worker. Use it to spread jobs, such as resizing images or sending emails.
- Publish/subscribe. Each event is delivered to every interested subscriber. Use it to announce that something happened, such as "order placed", so payments, inventory and email can each react.
Most real systems use both.
The rules that keep it reliable
1. Expect messages to arrive more than once
Brokers generally promise at-least-once delivery. A worker might crash after doing the work but before confirming it, so the message is delivered again. Your handlers must therefore be idempotent: running them twice has the same effect as running them once.
Outbox pattern: save the event with the order, publish after.
def handle_payment(event):
if payments.exists(idempotency_key=event.id):
return # already processed, safely ignore
charge_card(event)
payments.record(idempotency_key=event.id)
2. Retry with backoff
Temporary failures are normal. Retry after a short delay, then a longer one, adding a little randomness so retries do not all hit at once. Cap the number of attempts.
3. Use a dead-letter queue
After the last retry, move the message aside instead of discarding it or blocking the queue. Alert when the dead-letter queue is not empty, and keep tooling to inspect and replay messages.
4. Do not lose events between the database and the queue
A classic bug: you save the order, then publish the event, and the process crashes in between. The order exists and nobody hears about it. The outbox pattern fixes this: write the event to an outbox table in the same database transaction as the order, and have a separate process publish it. Either both happen or neither does.
5. Think about ordering
Most queues do not guarantee global order. If order matters for a particular customer or account, route all their messages to the same partition or queue, and design handlers to cope with late arrivals.
6. Version your messages
Events outlive the code that wrote them. Include a version and add fields rather than renaming them, so old and new consumers can coexist.
Which broker to choose
| Tool | Good fit |
|---|---|
| Amazon SQS or a similar managed queue | Simple background jobs, minimal operations |
| RabbitMQ | Flexible routing, task queues, many languages |
| Redis-based queues (for example BullMQ or Sidekiq) | Small to mid-size job processing when you already run Redis |
| Apache Kafka | High-volume event streams, replay, many independent consumers |
| Cloud pub/sub services | Fan-out between services on a single cloud |
Start with the simplest thing that fits. Teams often reach for Kafka when a managed queue would do, and pay for it in operations.
Start with the simplest broker that fits.
What queues cost you
- Eventual consistency. The email goes out a moment after the order, not at the same instant. The interface must be honest about that, for example showing "payment processing".
- Harder debugging. A request now spans several services over time, so you need request IDs and tracing.
- More moving parts to monitor: queue depth, consumer lag, retry counts.
When not to use one
- The caller genuinely needs the result immediately, such as a price lookup.
- Your system is small, and a single process with a database transaction is simpler and safer.
- The team has no capacity to monitor another component.
Do not introduce a broker because it sounds modern. Introduce one when a slow or unreliable step is hurting users.
Conclusion
Queues let your system stay fast and calm when parts of it are slow or broken. Use them for work that can happen after the response, keep handlers idempotent, retry with backoff, send failures to a dead-letter queue and guard the link between your database and your broker with an outbox. Start small, and add complexity only where it pays for itself.