In an event-driven architecture (EDA), services react to events instead of calling each other directly. An event is a record that something happened: "order placed", "payment failed", "file uploaded". The service that emits it does not know who reacts.
Publish/subscribe (pub/sub) is the messaging pattern that makes this work. This note covers how pub/sub delivers messages, what happens when a consumer fails, and how to stay correct when messages arrive twice.
Event-driven style#
In a request-driven design, the order service calls the email service, the stock service and the analytics service one by one. It must know all three, and it waits on each.
In an event-driven design, the order service publishes one OrderPlaced event. Each interested service subscribes and reacts on its own time. That one change has four effects:
- Loose coupling. You add a new consumer without touching the publisher.
- Independent scaling. Publishers and consumers scale separately.
- Resilience. If the email service is down, events wait for it. The order still succeeds.
- Cost. The flow is harder to follow, and the system becomes eventually consistent.
Because an event records something that already happened, name it in the past tense, like OrderPlaced. It describes a fact, not a command.
Point-to-point vs pub/sub#
Both styles use a broker between senders and receivers. They differ in who gets each message.
| Point-to-point (queue) | Pub/sub (topic) | |
|---|---|---|
| Who gets a message | One consumer | Every subscription |
| Typical use | Task queue, work distribution | Broadcast events to many services |
| Adding a reader | Shares the existing work | Gets its own copy of every event |
| Examples | Amazon SQS, RabbitMQ queue | Google Cloud Pub/Sub, Amazon SNS, Kafka topics |
Queues covers buffering, backpressure and partitions. This note focuses on the pub/sub side.
The parts of a pub/sub system#
Whichever style you use, the same named pieces come up in every broker's docs.
| Part | What it is |
|---|---|
| Publisher (producer) | Creates messages and sends them to a topic |
| Message | The data plus metadata (ID, attributes, timestamp) |
| Topic | A named feed that publishers write to |
| Subscription | A named interest in a topic; it tracks which messages its readers have acknowledged |
| Subscriber (consumer) | Receives messages from one subscription |
| Broker | The service that stores messages and routes them to subscriptions |
| Schema (optional) | Rules for the message format, checked on publish |
The key idea: a topic copies each message into every subscription. Inside one subscription, each message goes to one of its subscribers.
Message lifecycle#
A broker does not forward and forget. It persists each message so that a slow or crashed subscriber does not lose it.
- A publisher sends a message to a topic.
- The broker writes the message to durable storage, usually replicated across zones.
- The broker makes the message available to every subscription on the topic.
- Each subscription delivers it to one of its subscribers.
- The subscriber processes the message and sends an ack.
- Once every subscription has an ack, the broker can delete the message.
How long an unacknowledged message stays depends on the service. Pull-based brokers keep it until a retention period ends. Push-only notification services keep it for a bounded retry window, then drop it or move it to a dead letter queue. Kafka works differently: it keeps messages for a set time whether or not anyone has read them, and each consumer group tracks its own position.
Ack, nack and redelivery#
Persisting a message is only half the job. The broker also needs to know whether the subscriber handled it, so each delivered message ends in one of three states.
- Acked. The subscriber finished the work and told the broker. The broker will not deliver it again to that subscription.
- Unacked. No ack arrived before the deadline. The broker assumes the subscriber failed and delivers the message again.
- Nacked. The subscriber says "I cannot handle this now". The broker redelivers it right away, or after a backoff delay if the subscription has a retry policy.
Ack deadline (visibility timeout)#
The broker can't wait forever for an ack, so it gives each subscriber a time limit. When a subscriber receives a message, the broker hides it from other subscribers for a while. Google Cloud Pub/Sub calls this the ack deadline. Amazon SQS calls it the visibility timeout.
- If the subscriber acks in time, the message is done.
- If the deadline passes first, the message becomes visible again and another subscriber may get it.
- If work takes long, extend the deadline while you process. Otherwise a second worker starts the same job.
A late ack, a lost ack, or a crash after the work but before the ack all lead to the same message delivered twice. This is normal, not a bug.
Pull vs push delivery#
So far the subscriber has "received" a message without saying how. In practice it gets messages in one of two ways.
| Pull | Push | |
|---|---|---|
| How | Subscriber asks the broker for messages | Broker sends an HTTP request to the subscriber |
| Ack | Explicit ack call | A success response (for example 2xx) |
| Flow control | Subscriber controls the rate | Broker controls the rate |
| On failure | Message reappears after the deadline | Broker retries with backoff |
Push delivery is not fire-and-forget. If the endpoint fails or times out, the broker retries, usually with exponential backoff, for a limited time. After that, the message goes to a dead letter queue if you set one up, or it is dropped. Some endpoint types, such as email or SMS in notification services, get few or no retries, so do not rely on them for critical work.
Fan-in, competing consumers and fan-out#
Topics and subscriptions combine into three common shapes.
- Fan-in (many to one). Many publishers write to one topic. One subscription and one subscriber read everything. Example: many services send audit events to one audit log writer.
- Competing consumers (load balanced). One subscription has several subscribers. Each message goes to only one of them. Add subscribers to process more messages per second.
- Fan-out (one to many). One topic has several subscriptions. Each subscription gets every message. Use this when different services need to do different work with the same event.
You can combine the last two. Each subscription in a fan-out can have its own pool of competing workers.
A common cloud example of fan-out is a notification topic (Amazon SNS) that delivers into several queues (Amazon SQS). Each queue then holds its copy until its own workers process it.
Two subscriptions that point at the same endpoint each deliver every message. The endpoint sees every message twice.
Filtering#
Many brokers let a subscription set a filter on message attributes, such as event_type = "OrderShipped". The subscription then receives only matching messages. Without a filter, every subscription gets every message on the topic.
Dead letter queues#
Some messages fail every time: bad data, a missing record, a bug. If the broker keeps redelivering them, they waste capacity and can block other work.
A dead letter queue (DLQ) catches them. You set a maximum number of delivery attempts. After that, the broker moves the message to the DLQ instead of retrying again.
- Alert when the DLQ is not empty.
- Inspect the messages, fix the cause, then replay them.
- Keep enough data in the message (or its attributes) to debug it later.
Both pull and push systems support this. Google Cloud Pub/Sub calls it a dead-letter topic.
Delivery guarantees and idempotency#
Redelivery keeps messages from getting lost, but it's also why duplicates show up. Most brokers promise at-least-once delivery. Every message arrives, but some arrive more than once. Retries, late acks and redelivery after a crash all cause duplicates.
| Guarantee | Meaning | Cost |
|---|---|---|
| At-most-once | Never duplicated, may be lost | Loses data on failure |
| At-least-once | Never lost, may be duplicated | Consumer must be idempotent |
| Exactly-once processing | Effect happens once | Limited scope, lower throughput |
"Exactly-once" in a managed queue has a narrow meaning. Amazon SQS FIFO queues, for example, drop a message whose deduplication ID they already saw within a deduplication window (five minutes). A duplicate sent after the window, or a consumer that crashes mid-task, can still cause double processing. So FIFO gives exactly-once processing within the window, not a guarantee across your whole system.
The safe default: make every consumer idempotent. Processing the same message twice must have the same effect as processing it once. Here's what that looks like for a payment handler:
def handle(message: dict, db) -> None:
"""Charge an order once, even if the message arrives twice."""
event_id = message["event_id"]
with db.transaction():
# A unique constraint on event_id rejects a second insert.
if not db.try_insert("processed_events", event_id=event_id):
return # already handled; ack again and move on
db.charge(order_id=message["order_id"],
amount=message["amount"])Record the event ID and apply the effect in one transaction. If they are separate steps, a crash between them breaks idempotency.
Ordering#
Duplicates aren't the only surprise. Pub/sub systems don't order messages across a whole topic by default.
Where order matters, give related messages the same ordering key (Pub/Sub) or message group ID (SQS FIFO), or use a Kafka partition key. The broker then keeps order within that key only. Strict order also means a failed message holds back the ones behind it with the same key.
Trade-offs and pitfalls#
Event-driven systems trade simple call chains for flexibility.
- Hard to trace. One user action fans out into many asynchronous steps. Pass a correlation ID (one ID shared by every message from the same user action) in every message and use distributed tracing.
- Eventual consistency. Readers may see old data for a short time. Design screens and APIs to allow it.
- Schema changes. Many consumers read the same event. Add fields; do not remove or rename them without a migration plan.
- Poison messages. A message that fails every time is called a poison message. Without a DLQ, one bad message can loop forever.
- Hidden backlog. Watch the age of the oldest unacked message per subscription.
When to use it#
Pub/sub fits when one event matters to several independent services, or when the publisher should not wait for the work to finish.
- Good fits: notifications, audit logs, search indexing, analytics feeds, cache invalidation, starting background jobs.
- Poor fits: requests where the caller needs an immediate answer, such as checking a password. Use a synchronous call for those.
Key takeaways#
- Pub/sub copies each message to every subscription; competing consumers on one subscription share the work.
- Brokers persist messages and redeliver any message that is not acked before its deadline.
- Push delivery retries with backoff; failed messages end up in a dead letter queue if you configure one.
- Expect duplicates. Make consumers idempotent by recording the event ID in the same transaction as the effect.
- FIFO "exactly-once" means deduplication within a time window, not a system-wide guarantee.