Regionalization and Latency

Distance is delay. No amount of code makes light travel faster through fibre, so a request from London to a server in Virginia always pays tens of milliseconds each way. To be fast globally you move servers and data closer to users. And because a global system has many moving parts, you also need patterns for when calls between them fail.

Data locality#

Performance is best when the user, the compute and the data are physically close to each other.

• Regional deployment: keep web servers and their database in the same region (for example us-east). If servers are in New York and the database is in Los Angeles, every query pays a cross-country round trip of roughly 60 ms.
• Availability zones (AZs): separate data centres in the same metro area, linked by fast private fibre (around a millisecond apart). Spreading across AZs means one building losing power doesn't take you down, without a noticeable latency cost.
• Multi-region: run copies of your stack in several regions (say London, Tokyo and New York) and send each user to the nearest one. Replicate data into those regions so a London user isn't waiting on a database in New York just to load their profile.

Content delivery networks (CDNs)#

A CDN is a global network of edge servers in hundreds of cities. Instead of every user connecting to your origin (your own servers), they connect to a nearby edge location.

• First request: the edge doesn't have the file, so it fetches it from the origin and caches a copy.
• Next requests: later users in that city get the file straight from the edge, in a few milliseconds instead of a long trip to the origin.
• Dynamic content: for requests that can't be cached, edges keep warm, pre-established connections to the origin and route over optimised paths. CDN vendors often call this dynamic site acceleration.
• Origin shield: an extra caching tier between edges and origin, so a cache miss in many cities at once becomes a single request to your origin rather than a stampede.

What to put on a CDN:

• Static assets: images, CSS, JavaScript, video. Video platforms rely heavily on CDNs.
• Slow-changing data: product catalogues, public profiles, popular search results.
• Anything for a global audience far from your main data centre.

Benefits: lower round-trip time, far less load on your origin (it no longer serves the same logo millions of times), and some resilience, because edges can keep serving cached content during a brief origin outage.

Stale content: if you update a file but the CDN still has the old copy, users see old content. Fix it with versioned file names (style.v2.css, image_v2.png) so a new version is a new URL, or by purging the cache.

Regional partitioning#

When data only matters locally, you can split the whole system into independent geographic "islands", sometimes called cells.

• The idea: a rider in Miami never needs to see drivers in Seattle. The data has natural geographic boundaries.
• Each region gets its own stack: load balancers, app servers and databases, all in a data centre near its users.
• Good fits: ride-sharing (trips are local to a city), food delivery (restaurants within a few miles), local marketplaces.

Benefits:

1. Speed: both user-to-server and server-to-database trips are short.
2. Fault isolation: if one region's data centre fails, other regions are untouched.
3. Scalability: ten moderate databases are easier to run than one enormous global one.

The traveller problem: a Miami user flies to London. A global entry point, such as geo-aware DNS, routes them to the London cell, and you decide how their home-region data (profile, payment methods) follows them.

Handling failures#

In a distributed system the network is unreliable. Calls will fail, hang or return errors, and your design must expect it.

1. Retries with exponential backoff and jitter#

For transient failures (a brief network blip, a momentary overload), retry.

• Exponential backoff: wait longer after each failure (1 s, 2 s, 4 s, 8 s…) instead of hammering a struggling server every second.
• Jitter: add randomness to each wait. Without it, thousands of clients that failed at the same moment retry at the same moment too, a thundering herd that acts like a self-inflicted denial-of-service attack.
• Cap the number of attempts and the maximum delay.

import random
import time


def call_with_retries(fn, max_attempts=5, base=0.5, cap=8.0):
    for attempt in range(max_attempts):
        try:
            return fn()
        except ConnectionError:
            if attempt == max_attempts - 1:
                raise
            # "Full jitter": sleep a random time up to the
            # exponential bound, so clients that failed
            # together don't retry together.
            bound = min(cap, base * 2 ** attempt)
            time.sleep(random.uniform(0, bound))

2. Idempotency keys#

Retrying is dangerous when the call has side effects. Retry a payment and you might charge the customer twice. The fix is to make the operation idempotent: doing it twice has the same effect as doing it once.

1. The client generates a unique idempotency key (a UUID) for the operation and sends it with every attempt.
2. The server checks a fast store (often Redis or a database table) for that key.
3. Not seen: process the request, then store the key with the result.
4. Seen and finished: return the stored result without processing again.
5. Seen but still in progress: reject or wait, so two attempts don't run at the same time.

This makes writes as safe to retry as reads.

3. Circuit breakers#

A cascading failure happens when one slow service makes its callers wait, their threads fill up, and they become slow too, until the whole system is stuck. A circuit breaker stops calling a service that is clearly failing.

1. Closed (normal): requests flow. The breaker counts errors.
2. Open (tripped): once errors pass a threshold (say 50% over a window), every call fails fast without touching the struggling service, giving it time to recover.
3. Half-open (testing): after a timeout (say 30 s), a few trial requests are allowed. If they succeed, the breaker closes. If they fail, it opens again.

What you do while the circuit is open depends on the operation:

Summary#

Used together: retry transient errors with backoff and jitter, protect side effects with idempotency keys, and wrap each dependency in a circuit breaker with a sensible fallback.

Key takeaways#

• Latency is bounded by physical distance, so keep users, servers and data close together.
• Use multiple AZs for resilience within a region and multiple regions for global users.
• CDNs cache static and slow-changing content near users. Use versioned URLs to avoid stale files.
• Regional partitioning (cells) gives low latency and fault isolation when data is naturally local.
• Retries need exponential backoff with jitter, and side-effecting calls need idempotency keys.
• Circuit breakers stop one failing dependency from taking down everything that calls it.