Load Balancing

When one server can't handle your traffic, you add more servers. That only helps if each request knows which server to go to. A load balancer makes that decision: it spreads requests across a pool of servers, skips the ones that are unhealthy, and lets you add or remove servers without clients noticing. This lesson covers how scaling works, the two families of load balancers, and how to pick an algorithm.

Vertical vs horizontal scaling#

There are two ways to add capacity: a bigger machine or more machines.

• Vertical scaling (scale up): give one machine more CPU, RAM or faster disks. If one server handles your load today and you need double, buy a bigger server. Modern hardware is very capable, so this goes further than people expect.
• Horizontal scaling (scale out): add more identical machines and split the work between them.

A typical path: scale vertically first because it is quickest. When you approach hardware limits or need redundancy, scale horizontally. One way to put it: keep scaling up until the next upgrade costs more than running a cluster.

One caveat: horizontal scaling assumes servers are interchangeable. If a server keeps user state in its own memory, sending that user to a different server breaks things. Either move that state into a shared store (such as Redis) or use sticky routing (covered under algorithms below).

Why load balancers#

A load balancer gives you four things at once.

• Scalability: add servers as traffic grows.
• Availability: if one server fails, the others take its traffic.
• Performance: even distribution keeps response times low.
• Maintainability: take servers out for deploys or maintenance without downtime.

There are two places the balancing decision can happen: in the client or in a dedicated balancer in the middle.

Client-side load balancing#

The caller picks which server to talk to itself, using a "phone book" instead of going through a middleman.

How it works#

1. Service registry: a directory such as Consul or Eureka keeps a live list of healthy server addresses for each service. Run several registry instances that replicate the list so the registry isn't a single point of failure.
2. Fetch and cache: on startup the client downloads the list and keeps it in local memory.
3. Pick: for each request, a small library inside the client (for example gRPC's built-in picker) chooses an address from the cached list, using round robin or similar, and calls that server directly.

The client doesn't call the registry per request. That would add back the extra hop it is trying to avoid.

Keeping the list fresh#

With polling there is a stale-data gap: if a server dies just after a poll, clients keep sending it traffic until the next refresh.

Handling the stale gap with passive health checks#

However fast the registry is, the client's copy can be briefly wrong. Clients cover this themselves:

1. The client calls an address and the connection fails or times out.
2. It immediately marks that address as bad in its own memory.
3. It retries the request on another address from the cache.

The system heals in milliseconds, before the registry's update arrives. The full recovery after a crash looks like this:

1. A server crashes.
2. Clients hit it, mark it bad locally and retry elsewhere. Users see no errors.
3. The registry's health check fails and it removes the server from the master list.
4. The orchestrator (an autoscaling group or Kubernetes) sees too few instances and starts a new one.
5. The new server registers its address.
6. Clients pick up the new address through a push or the next poll.

When to use it#

• Internal microservices with high-frequency service-to-service calls.
• gRPC systems, where long-lived HTTP/2 connections can defeat a simple central balancer.
• Very high throughput backends where a central balancer would become a bottleneck.

Pros: no extra network hop, no central bottleneck (balancing work is spread across all clients), and no single point of failure (if the registry is down, clients keep using their cached list).

Cons: "thick" clients with more complex code; possible stale lists; and internal addresses are exposed to whoever runs the client.

DNS is client-side balancing too#

When you look up a domain, the DNS server can return several IP addresses and rotate their order on each lookup. Different clients get different orderings and so connect to different servers. DNS is effectively doing client-side load balancing. The catch is that DNS answers are cached, so removing a dead server takes a while to reach every client.

Dedicated load balancers#

A dedicated balancer is a middleman, hardware or software, that is the single entry point for clients. Every request takes an extra hop: client to balancer, balancer to server. In return you get instant updates to the server list and fine-grained control over routing. There are two kinds, named after the OSI layer they work at (see Protocols: Layer 4 and 7 for what each layer can see).

Layer 4 load balancer#

"Simple and fast." It routes on IP address and port only, without reading the contents.

• It forwards a TCP (or UDP) connection to one backend server, and everything on that connection goes to the same server.
• It doesn't inspect the payload, so per-request CPU cost is low.
• Use for: long-lived connections such as WebSockets; performance-critical paths; non-HTTP traffic like database connections or custom game protocols.
• On AWS this is the Network Load Balancer.

Layer 7 load balancer#

"Smart but heavier." It understands HTTP: URLs, headers and cookies.

1. It terminates the client's connection and opens a separate connection to the chosen server.
2. It reads the request.
3. It routes based on what's being asked for.

• Use for: ordinary web traffic and REST APIs; path-based routing (/api to one pool, /images to another); routing on headers or cookies.
• Bonus features: can act like a simple API gateway; can block malicious requests (WAF); handles TLS termination so app servers don't decrypt. See SSL variants.
• On AWS this is the Application Load Balancer.

Examples by scenario#

Layer 4:

• Real-time games using UDP: the balancer forwards packets with almost no delay because it doesn't try to read game data.
• Database pools: you only need a stable TCP connection between app and database.
• Voice and video (SIP, RTP): raw packet throughput matters more than content inspection.

Layer 7:

• Microservices: /api/users to the user service and /api/payments to the payment service, behind one public IP.
• A/B tests and canary releases: send 10% of users to a new version based on a header or cookie.
• TLS termination: decrypt at the balancer and send plain HTTP to backends.
• Web application firewall: because it sees content, it can block SQL injection or cross-site scripting attempts.

Both together. Large systems often put an L4 balancer at the very front to absorb millions of raw connections and spread them across a pool of L7 proxies (such as nginx or a Kubernetes ingress), which then do the smart routing, auth and caching. A common pattern is L7 at the edge for TLS and routing, and L4 for internal hops where you want minimal overhead.

WebSockets work through either type. L4 is a natural fit because it just forwards long-lived connections, but most modern L7 balancers support WebSockets too.

Health checks and fault tolerance#

The balancer regularly checks each server to see if it can still do work.

1. Interval: how often to check, e.g. every 5 seconds.
2. Threshold: how many failures before a server is marked unhealthy, e.g. 3 in a row.
3. Removal: an unhealthy server stops getting new traffic immediately.
4. Recovery: the balancer keeps checking, and after enough passes the server goes back into rotation.

Why it matters: failover is automatic, users don't notice when one server dies, and servers that recover from a brief overload rejoin on their own.

Trade-offs:

• Frequent deep checks cost CPU and bandwidth on the backends.
• Zombie servers pass a TCP check but fail every real request with a 500. This is why HTTP checks are preferred for important services.
• A common design: each service exposes /health, the L7 balancer calls it every few seconds, and the endpoint also checks the service's own database and cache connections, so a server only takes traffic when its dependencies are reachable. Be careful with this: if a shared database blips, every server fails its check at once.
• Combine active checks with passive ones: if a real request fails, mark the server bad and retry elsewhere straight away.

Load balancing algorithms#

The algorithm decides which server gets the next request.

• Default to round robin or random for stateless services. New servers start receiving their share automatically.
• Least connections for persistent connections. With round robin, long-lived connections slowly pile up on older servers. Least connections sends new ones to the emptiest server, so new servers immediately take load.
• IP hash only when you must. If a user's data lives only in one server's local memory, you need to keep that user on that server. Watch out for hotspots: thousands of users behind one office or carrier IP all land on the same server.

Scaling WebSockets: the pub/sub backplane#

Users A and B are in the same chat room, but the load balancer connected them to different servers. Server 1 only knows about its own connections, so it can't deliver A's message to B directly.

The fix is a message backplane: a shared broker, often Redis Pub/Sub, that every server subscribes to.

1. User A sends a message to server 1.
2. Server 1 sees it's for room 101 and publishes it to the Redis channel room_101.
3. Every server with a member of room 101 is subscribed to that channel.
4. Redis delivers the message to all of those servers.
5. Server 2 checks its local connections, finds user B, and pushes the message down B's WebSocket.

Why this works well:

• No server tracking: server 1 doesn't need to know where B is connected.
• Horizontal scaling: add 100 servers. As long as they subscribe to the right channels, rooms keep working.
• Disposable servers: if server 1 dies, user A reconnects (clients must have automatic reconnection logic), lands on another server, and that server subscribes to the same channel.

Key takeaways#

• Scale vertically first for simplicity, then horizontally for capacity and redundancy.
• Client-side balancing (registry plus cached list) removes a hop and suits internal services. Never use it for public clients.
• Dedicated L4 balancers forward connections cheaply; L7 balancers read HTTP to route by path, header or cookie.
• Use deep HTTP health checks plus passive checks so dead or zombie servers are removed quickly.
• Default to round robin; use least connections for long-lived connections; avoid IP hash by making servers stateless.
• Scale WebSockets with a pub/sub backplane so servers can deliver messages to users connected elsewhere.