Message queues and publish/subscribe

Messaging lets systems communicate without being available at the same time. A sender puts a message on a broker and moves on; receivers pick it up when they’re ready. There are two basic shapes:

• Point-to-point (queue): each message is processed by exactly one consumer. Used to distribute work.
• Publish/subscribe (topic): each message is delivered to every subscriber. Used to broadcast that something happened.

Diagram

flowchart LR
    subgraph q [Point-to-point]
        p1[Order API] --> queue[[Work queue]]
        queue --> w1[Worker 1]
        queue --> w2[Worker 2]
    end
    subgraph ps [Publish/subscribe]
        p2[Order API] --> topic{{OrderPlaced topic}}
        topic --> s1[Billing]
        topic --> s2[Shipping]
        topic --> s3[Analytics]
    end

How it works

• Brokers (such as RabbitMQ, Amazon SQS/SNS, Azure Service Bus or Apache Kafka) store messages until they’re consumed.
• Acknowledgement. A consumer confirms a message only after processing it; unacknowledged messages are redelivered.
• Delivery guarantees. Most systems offer at-least-once delivery, so duplicates are possible.
• Dead-letter queues collect messages that repeatedly fail, so they can be inspected without blocking others.
• Queues vs event streams. Traditional queues delete messages once consumed. Event-streaming platforms such as Kafka keep an ordered log that many consumers can read and replay independently.

Pros

• Temporal decoupling: the receiver can be down or slow without failing the sender.
• Load leveling: queues absorb bursts and let consumers work at their own pace.
• Easy scaling: add consumers to process more messages in parallel.
• Loose coupling: publishers don’t need to know who subscribes.

Cons

• No immediate answer: the sender doesn’t learn the outcome right away.
• Duplicates and ordering: consumers must handle redelivery and, often, out-of-order messages.
• Another system to run and monitor, with its own failure modes.
• Harder tracing: following one business transaction across queues needs correlation IDs.

When to use it

• Background work that doesn’t need to finish before the user gets a response.
• Integrating systems with different availability or throughput.
• Broadcasting business events to several downstream systems.

When to avoid it

• The caller needs an immediate result to continue, such as a fraud check before approving a payment.
• Very simple systems where a direct call is clearer and the dependency is reliable.

Common pitfalls

• Assuming exactly-once delivery. Design consumers to be idempotent.
• Huge messages. Put large payloads in storage and send a reference.
• No dead-letter handling, so one bad message blocks a queue or retries forever.
• Publishing before the database commit, or committing without publishing. Use the transactional outbox pattern.

Related patterns

• Event-driven architecture
• Enterprise service bus
• Transactional outbox
• Competing consumers

Glossary: at-least-once delivery, transactional outbox

Further reading

• Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns
• Martin Kleppmann, Designing Data-Intensive Applications, chapter 11