Caching

A cache keeps a copy of frequently used data in fast memory so you don't have to go to slow storage (or redo expensive work) on every request. It is usually the first tool you reach for when a system is read-heavy or too slow. The hard part is not adding a cache; it's deciding what goes in it, how writes interact with it, and how stale data gets cleaned up.

Why caching works#

Your database keeps data on disk. Memory is orders of magnitude faster:

End to end, a query that takes 5–50 ms against the database often takes 0.1–1 ms from an in-memory cache like Redis.

Hits, misses and hit ratio#

• Cache hit: the data is in the cache. Return it; the database is never touched.
• Cache miss: the data isn't there. Read it from the database, store it in the cache, then return it. Slow this once, fast after that.

$\text{hit ratio} = \frac{\text{cache hits}}{\text{total requests}}$

The hit ratio tells you whether the cache is earning its keep. With hits at 1 ms and misses at 50 ms:

As a rule of thumb, above 90% is healthy and above 99% means the database is almost never touched.

What to cache (and what not to)#

A useful trick for expensive work: cache by content hash. If the same input arrives many times (for example, identical text sent to an expensive ML model), the first request computes the result and every repeat is a cache hit, saving both latency and compute cost.

Where caches live#

Caching isn't one thing. It's a stack of layers, each closer to the user and each shielding the layer behind it.

Browser caching is controlled with the Cache-Control header:

Together these layers can absorb most traffic. In a typical setup, static assets are served by browsers and the CDN, Redis answers most dynamic reads, and only cache misses reach the database.

For CDN details (pull vs push, invalidation), see Scaling Reads.

Caching strategies#

These strategies decide how data gets into the cache and what happens on a write.

Cache-aside (lazy loading): the default#

The application owns the logic. On a read it checks the cache first and fills it on a miss. On a write it updates the database and deletes the cache key.

Why delete instead of updating the cache on a write? Deleting keeps the database as the single source of truth: the next read misses and loads fresh data. Updating means keeping two copies in sync, and if the cache update fails you serve stale data.

def get_user(user_id):
    key = f"user:{user_id}"
    cached = redis.get(key)
    if cached is not None:
        return cached  # hit: database untouched
    # Miss: go to the source of truth.
    user = db.query_user(user_id)
    # The TTL is a safety net for missed invalidations.
    redis.set(key, user, ex=3600)
    return user


def update_user(user_id, fields):
    # Write the source of truth first.
    db.update_user(user_id, fields)
    # The next read reloads fresh data.
    redis.delete(f"user:{user_id}")

• Pros: only caches what is actually requested; if the cache dies the system still works (just slower); simple.
• Cons: the first request for each key is slow; data can be stale between a write and the next read.

Read-through#

Same idea, but the cache fetches from the database on a miss. Your code only talks to the cache. App code is simpler, but the cache must know how to load from your database. Amazon DynamoDB Accelerator (DAX) works this way.

Write-through#

Every write goes to the cache and the database, and is confirmed only after both succeed. Reads always see fresh data, but writes are slower and the cache fills with data that may never be read. Good when data is read right after it's written (a user updates their profile, then views it).

Write-back (write-behind)#

Writes go to the cache only and are acknowledged immediately; the cache flushes to the database later in batches. This gives the fastest writes and fewer DB round trips, but if the cache crashes before flushing, those writes are lost, even though the user was told they succeeded. Use it only for data you can afford to lose, like view or like counts. Never for orders or money.

Write-around#

Writes go straight to the database and skip the cache; reads fill the cache on a miss. This keeps write-once, rarely-read data (like log entries) from polluting the cache, at the cost of a miss on the first read after a write.

Choosing a strategy#

Cache invalidation#

When the source data changes, the cache may still hold the old value. Keeping them in sync is famously one of the hardest problems in computer science.

TTL (time to live)#

Every entry expires after a fixed time, e.g. SET user:123 ... EX 300. The next read after expiry reloads from the database.

• Short TTL (30 s): less staleness, more database load. Long TTL (1 h): the reverse.
• Dead simple and self-cleaning, but data can be stale for up to the TTL.
• Good when slight staleness is fine: product listings, news articles.

Event-driven invalidation#

On every write, delete the key right away (as in cache-aside). The stale window is close to zero, but every write path must know about the cache, and in a distributed system the delete itself can get lost. Good when freshness matters: inventory counts, account balances.

1. A reader misses and loads the old value from the database.
2. Meanwhile a writer updates the database and deletes the key.
3. Then the reader writes its old value into the cache, and the stale value sticks around.

One fix is a delayed double delete: write to the DB, delete the key, wait briefly (say 500 ms), and delete it again to catch stale values written by in-flight readers.

Version-based keys#

Put a version in the key: user:123:v5. When the record changes, the version becomes v6, so lookups for user:123:v6 miss and load fresh data, while the old v5 entry quietly expires via TTL. There's no race on a key that's already been invalidated, but you have to track versions.

The same trick works for bulk invalidation: include a policy or schema version in the key (result:{hash}:policy:v3), bump it, and every old entry stops matching.

What production systems usually do#

Combine TTL + event-driven deletes. Deletes keep data fresh in the normal case, and a TTL (say one hour) is the safety net if an invalidation is lost, so stale data never lives forever.

Eviction policies#

A cache has limited memory. When it's full, the eviction policy decides what to throw out.

• LRU (least recently used): evict whatever hasn't been touched for the longest time. The assumption is that recently used data will be used again soon. Works for most workloads and is the common default. Weakness: a one-off batch job that scans thousands of old records makes them all "recently used" and pushes out genuinely hot data.
• LFU (least frequently used): evict whatever has the lowest access count. It resists batch-scan pollution, but a new trending item starts with a low count and can be evicted, while old items that were once popular hang around.
• FIFO: evict the oldest insert, ignoring access patterns. Simple but usually worse.
• Random: evict a random entry. Almost zero overhead and surprisingly decent.

Example: a news site's cache is full and holds the homepage (very hot), a breaking story (new, 500 hits in 10 minutes) and a 2020 article (2 hits today). LRU evicts the 2020 article, which is right. LFU might evict the breaking story because its total count is still low. That's why recency is usually the better signal.

TTL is not an eviction policy. It removes entries based on age, for freshness. The common setup is LRU for space plus TTL for freshness (in Redis, maxmemory-policy allkeys-lru with a TTL on every key).

Caching in an interview#

Walk through these questions in order:

1. Should we cache this? Is it read-heavy and slow to change?
2. Which strategy? Cache-aside unless there's a reason not to.
3. How do we invalidate? TTL plus delete-on-write.
4. Which eviction policy? LRU with TTLs.
5. How do we distribute it? A Redis cluster with consistent hashing. See Distributed Cache.
6. Is there a stampede risk? Use lock-on-miss for hot keys.

Key takeaways#

• Caches trade freshness for speed: memory is far faster than disk, and the hit ratio tells you whether the cache is working.
• Cache-aside is the default: read from the cache, fall back to the DB on a miss, and on a write update the DB and delete the key.
• Write-back is fastest but can lose acknowledged writes. Keep it for data you can afford to lose.
• In production, combine event-driven deletes with a TTL safety net.
• LRU plus TTL is the usual eviction setup, because recency predicts reuse better than total frequency.