Amazon SNS and SQS

Amazon SQS (Simple Queue Service) and Amazon SNS (Simple Notification Service) are AWS's two basic managed messaging services. SQS is a queue: producers leave messages, and consumers pull them when they are ready. SNS is a pub/sub topic: publishers send one message, and SNS pushes a copy to every subscriber. Most event-driven designs on AWS use one or both, often together.

The general ideas behind them are covered elsewhere. Queues explains buffering, retries, idempotency and backpressure. Pub/sub and event-driven architecture explains topics, fan-out and loose coupling. This note covers how SQS and SNS implement them.

SQS vs SNS at a glance#

Before the details, it helps to see how far apart the two services are. They solve different problems, and the table shows the main differences.

Amazon SQS#

Start with the queue. Its whole lifecycle is three calls: a producer calls SendMessage, and SQS stores the message redundantly. A consumer calls ReceiveMessage, processes the message, and then calls DeleteMessage. Until the consumer deletes it, the message is not gone.

This keeps producers and consumers independent. If the consumers are down or slow, messages wait in the queue. When traffic spikes, the queue absorbs it and consumers work through the backlog at their own pace.

Standard vs FIFO queues#

The first decision for any queue is its type. You choose it when you create the queue, and you can't change it later.

The table hides two FIFO details that matter in practice:

• Message group ID. Order is kept per group, not across the whole queue. Use one group per customer or per order. More groups means more parallel consumers.
• Deduplication ID. If SQS sees the same deduplication ID again within a five-minute window, it accepts the send but doesn't deliver the copy. You can set the ID yourself or let SQS hash the message body.

"Exactly once" covers duplicate sends within that window. It doesn't protect you from a consumer that crashes after doing the work but before deleting the message. Consumers should still be idempotent.

Visibility timeout#

The receive-then-delete flow works because when a consumer receives a message, SQS doesn't remove it. It hides the message from other consumers for the visibility timeout. If the consumer deletes the message in time, it is gone. If not, the message becomes visible again and another consumer can receive it.

Set the timeout longer than your normal processing time. A consumer that needs more time can extend it with ChangeMessageVisibility. The default is 30 seconds and the maximum is 12 hours.

Retention and delay queues#

The visibility timeout controls when a message reappears. Two more settings control how long it lives and when it first appears.

• Retention period: how long SQS keeps an unprocessed message. The default is 4 days; you can set 1 minute to 14 days.
• Delay queue: every new message stays hidden for a set time after it is sent, up to 15 minutes. On standard queues you can also set a delay on single messages (a message timer).

Short vs long polling#

So far the consumer just calls ReceiveMessage. That call can wait for messages or return straight away.

• Short polling returns at once, even with no messages. It queries only some of SQS's servers, so it can miss messages that exist, and empty responses still cost a request.
• Long polling waits up to 20 seconds for a message to arrive. Set WaitTimeSeconds on the call or ReceiveMessageWaitTimeSeconds on the queue.

Prefer long polling. It cuts empty responses and cost, and messages arrive with less delay. Here is a long-polling receive that waits up to 20 seconds for a batch of up to 10 messages:

aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/orders \
  --wait-time-seconds 20 \
  --max-number-of-messages 10

Dead-letter queues#

The visibility timeout brings a failed message back for a retry, but some messages fail every time. A dead-letter queue (DLQ) is a second queue that collects them. You attach it with a redrive policy that sets maxReceiveCount, and each receive counts toward it. When a message passes that count without being deleted, SQS moves it to the DLQ.

• A DLQ must be the same type as its source: FIFO with FIFO, standard with standard.
• Set the DLQ's retention longer than the source queue's, so moved messages don't expire too soon.
• Alarm on the DLQ's depth. A growing DLQ means a bug or bad data.
• After a fix, use DLQ redrive to move the messages back to the source queue.

Without a DLQ, a "poison" message that always fails keeps coming back until it expires.

Amazon SNS#

Where SQS waits for consumers to pull, SNS pushes. An SNS topic is a named channel, and publishers call Publish on it. Each subscription links the topic to one endpoint. SNS pushes a copy of every message to every subscription whose filter matches.

Publishers don't know who the subscribers are. You can add a new subscriber without touching the publisher.

Subscription types#

A subscription can point at many kinds of endpoint. SNS groups them into two families.

HTTP/HTTPS and email subscriptions must confirm the subscription before they receive messages. An SQS queue needs a queue policy that lets the topic send to it.

Filter policies#

Without a filter, every subscriber gets every message. A filter policy on a subscription lets it receive only the messages it cares about. By default the filter matches on message attributes. You can set it to match on the message body instead.

{
  "eventType": ["order_shipped"],
  "amount": [{ "numeric": [">=", 100] }]
}

With the policy above, the subscription receives only order_shipped events with an amount of 100 or more. Filtering in SNS saves each consumer from receiving and discarding messages it doesn't want.

Delivery retries#

Because SNS pushes, it has to decide what to do when an endpoint doesn't answer. SNS doesn't wait for a consumer to process a message, but it does retry when an endpoint is unavailable. Each protocol has a delivery policy.

For HTTP/S, SNS retries on 5xx and 429 responses. Other errors count as permanent and are not retried. When the retries run out, SNS discards the message, unless you attach a DLQ (an SQS queue) to the subscription.

Standard and FIFO topics#

Topics come in the same two types as queues. Standard topics give at-least-once delivery with best-effort order. FIFO topics keep order and remove duplicates, like FIFO queues. They deliver to SQS queues only, not to email, SMS or HTTP endpoints.

SNS also offers server-side encryption with AWS KMS, and IAM plus topic policies control who can publish and subscribe.

Fan-out: SNS to SQS#

Each service has a gap the other fills: SQS stores messages but delivers each to one consumer, and SNS copies to many but stores nothing. The most common pattern combines the two. You publish once to an SNS topic, and each consuming service has its own SQS queue subscribed to it.

1. The order service publishes OrderPlaced to the orders topic.
2. SNS pushes a copy to the billing, shipping and analytics queues.
3. Each service polls its own queue at its own pace.
4. If the shipping service is down, its messages wait in its queue. Billing isn't affected.

The queue gives each consumer storage, retries and a DLQ, which SNS alone can't give a pull-based consumer. To add a new consumer, subscribe a new queue. The publisher doesn't change.

When to use which#

Pick by how many consumers need each message.

Key takeaways#

• SQS is a pull-based queue that stores messages; SNS is a push-based topic that copies each message to every matching subscriber.
• Standard queues are at least once with best-effort order. FIFO queues keep order per message group and drop duplicates within the deduplication window.
• A received message stays hidden for the visibility timeout. Delete it after processing, or it comes back.
• Use long polling, and attach DLQs to queues and SNS subscriptions so failing messages aren't lost.
• SNS retries depend on the endpoint type. Filter policies keep subscribers from getting unwanted messages.
• SNS-to-SQS fan-out gives each consumer its own buffer, retries and DLQ.