Scaling Reads

Most applications read far more than they write, often 90% reads or more. When one database can't keep up, you scale reads in layers: copies of the database, caches in front of it, CDNs at the edge, data reshaped so reads are cheap, and specialized stores for queries the database is bad at. Each layer absorbs traffic so the next one sees less.

Read replicas#

Keep one leader (primary) that takes all writes, and add replicas that copy its data and serve reads.

Example: one database handling 1,000 writes/s plus 10,000 reads/s is overloaded. With three replicas, the leader handles the 1,000 writes and each replica serves about 3,300 reads/s.

How replication works: the leader executes a write and records it in a replication log (similar to a write-ahead log). Replicas stream that log and apply the same changes, so they end up with the same data, eventually.

Synchronous vs asynchronous replication#

Replication lag and read-after-write consistency#

With async replication, a replica can be a few milliseconds (or more) behind. A user places an order (written to the leader), immediately opens their order history (read from a lagging replica), and sees nothing. That's the read-after-write consistency problem, and interviewers like to ask about it.

Three fixes:

1. Read your own writes. After a user writes, route that user's reads to the leader for a short window (say 5 seconds). Everyone else keeps reading from replicas, so the extra load on the leader is small.
2. Monotonic reads. Pin each user to one replica, e.g. hash(user_id) → replica 2. Otherwise a user might see 100 orders from one replica and 95 from a lagging one, and data appears to vanish. With pinning, what they see only moves forward.
3. Strongly consistent reads for critical data. Order history can tolerate a bit of lag; an account balance should be read from the leader. Amazon DynamoDB exposes exactly this choice: eventually consistent reads (cheaper, may lag) vs strongly consistent reads.

Leader failure#

If the leader crashes, writes stop. Failover usually goes like this:

1. Replicas notice missing heartbeats.
2. The most up-to-date replica is promoted to leader.
3. Other replicas follow the new leader, and the app is pointed at it.

The risk: with async replication, a write the old leader confirmed but hadn't replicated yet is lost. Semi-synchronous replication reduces this. At least one replica must confirm before a write is acknowledged, so a promoted replica has the latest data.

How many replicas?#

Divide target read throughput by what one node handles: 100,000 reads/s ÷ 20,000 per node = 5 replicas. Then add headroom so losing one replica doesn't overload the rest, and consider a replica per region for latency. Three to five is a common starting point.

A cache in front of the replicas#

Put a cache like Redis in front of the database, so reads go cache → replica → never the leader:

1. Check Redis. On a hit, return in well under a millisecond.
2. On a miss, read from a replica (a few ms), store the result in Redis, and return it.

If 900 of 1,000 reads are cache hits, replicas see 100 and the leader sees none. See Caching for strategies and invalidation.

Cache warming: a brand-new, empty cache sends every request to the database at once. Before taking traffic, preload the most-accessed items (e.g. the top 1,000) so the cache starts warm.

CDN (content delivery network)#

If your servers are in Virginia, a user in Tokyo pays about 200 ms per round trip no matter how fast your code is. A CDN keeps copies of content on edge servers around the world, so each user is served from nearby (often ~20 ms).

What belongs on a CDN:

• Always: images, CSS, JS, video, fonts, PDFs. They're static, the same for everyone, and rarely change.
• Sometimes: public API responses and public pages (product listings, homepage), with a short TTL like Cache-Control: max-age=60.
• Never: personalized, authenticated or real-time data (your cart, your profile, live scores).

Invalidating CDN content:

1. Purge: call the CDN's API to drop an object from all edges.
2. Versioned URLs: publish logo.v3.png instead of overwriting logo.v2.png. A new URL is automatically a miss, and the old one is harmless. This is the cleanest approach.
3. Short TTL: content is never more than, say, a minute stale.

Denormalization#

A normalized schema stores each fact once, and queries JOIN tables to put the answer together. At scale those JOINs get expensive. Denormalization stores duplicate data so a read touches one row.

Normalized: showing an order joins 3 tables.

• orders(id, product_id, customer_id, quantity)
• products(id, name, price)
• customers(id, name, email)

Denormalized: showing an order reads 1 row.

• orders(id, product_id, product_name, product_price, customer_id, customer_name, quantity)

It's worth it when reads vastly outnumber writes and the duplicated data rarely changes. A product name copied into orders changes maybe once a year. It's not worth it when the copied value changes constantly, like an account balance.

Three levels, from lightest to heaviest:

1. Duplicate a column, e.g. product_name on orders.

2. Precompute aggregates. Store posts.like_count and increment it on each like instead of running COUNT(*) over millions of rows on every read. Large social apps keep counters like this.

3. Materialized views. Store a precomputed JOIN as a real table:

   CREATE MATERIALIZED VIEW order_summary AS
   SELECT o.id,
          p.name AS product,
          c.name AS customer,
          o.quantity * p.price AS total
   FROM orders o
   JOIN products p ON p.id = o.product_id
   JOIN customers c ON c.id = o.customer_id;

   Reads are instant, but the view is stale until you refresh it. That's fine for dashboards, not for real-time data.

Key-value and wide-column stores like DynamoDB have no JOINs, so they push you to design denormalized, self-contained items from the start. See Normalization vs Denormalization for the modeling side.

Search: use an inverted index#

Relational databases are built for structured lookups (WHERE id = 5), not free-text search. A query like WHERE name ILIKE '%wireless%' AND name ILIKE '%bluetooth%' can't use a B-tree index (because of the leading wildcard), so it scans the whole table, can't rank by relevance, and fails on typos ("wireles") or synonyms ("earbuds" vs "headphones").

A search engine like Elasticsearch uses an inverted index. Instead of "row → words", it stores "word → rows":

A search for "wireless bluetooth" intersects the two lists: $[1, 3] \cap [1] = [1]$.

Two lookups and an intersection, with no table scan. On top of that you get relevance scoring, fuzzy matching, synonyms, tokenization, and faceted search (filter by brand, price range or rating, like the sidebar filters on a shopping site).

Run it alongside your primary database. The database stays the source of truth; text searches go to Elasticsearch, and structured queries go to the database. Ways to keep the index in sync:

1. Dual writes: the app writes to both. Simple, but the app must handle partial failures.
2. Change data capture (CDC): stream the database's change log (e.g. PostgreSQL's WAL) into the index. More reliable.
3. Periodic re-index: rebuild every N minutes. Simple, but stale in between.

CQRS#

Command Query Responsibility Segregation takes the idea further and separates the write model from the read model entirely. Writes want normalized data, constraints and ACID. Reads want denormalized, pre-joined, fast data. One database can't be optimal for both.

The command side syncs its changes to the query side:

The cost is eventual consistency between the two sides, more infrastructure, and sync logic to maintain.

The typical scaling path#

Put together, the layers are usually added in this order.

Key takeaways#

• Replicas scale reads, but async replication causes lag. Know read-your-own-writes, monotonic reads, and reading from the leader for critical data.
• Put the cache in front of the replicas so the leader serves writes only. Warm new caches before sending them traffic.
• CDNs serve static, shared content from the edge. Prefer versioned URLs for invalidation.
• Denormalize when reads dominate and the copied data rarely changes.
• Use an inverted index for search, kept in sync via CDC. Reach for CQRS only when read and write needs truly diverge.