Be Wise Apps
Leer en español · Lire en français · Auf Deutsch lesen
Section 9.2 of 9.8 · Intermediate · 27 min read
Load balancing, statelessness and sessions
Video lesson
5 min · narrated, with captionsListen only (audio)
Read the transcript
Welcome to Section nine point two. Scaling out gives you many instances. This lesson is about what sits in front of them, and how to spread the work so that users never notice instances coming and going.
Picture a supermarket where a member of staff directs shoppers to tills. Sending people to each till in turn is fair but naive. One shopper with a trolley for a family of six holds up a till while the next is idle. Sending each shopper to the till with the shortest queue works much better. The director must also know which tills are open, and stop sending people to a till that is about to close, while its queue finishes. And it works best if any till can serve any shopper. That's the job of a load balancer.
A load balancer decides which instance receives each connection or request. It also checks which instances are healthy, gives clients one stable address, and often terminates TLS. A layer 4 balancer sees only IP addresses and ports. It picks a back end once per connection and forwards it without looking inside. That makes it extremely fast, and it works for any protocol. A layer 7 proxy understands HTTP. It reads each request, so it can route, retry, rate limit and log every one. Here it routes by path: order requests to the order pool, menu requests to the menu pool, and searches to the search pool. This difference catches many teams out. gRPC sends many requests over one long-lived connection. A layer 4 balancer chooses once per connection, so new instances can receive nothing. gRPC needs a layer 7 proxy, or balancing in the client.
Which instance should get the next request? Take two instances, A and B, each handling one request at a time. Eight requests arrive ten milliseconds apart: first a report taking a hundred milliseconds, then seven menu reads taking ten. Round robin sends requests in turn, whatever they cost. So three menu reads land on instance A, stuck behind the report. They take ninety, eighty and seventy milliseconds. The menu reads average about forty one milliseconds, the worst ninety. Meanwhile, instance B is idle half the time. Least connections sends each request to the instance with the fewest in progress. The report goes to A, and because B is always free again in time, every menu read goes to B. Now every menu read takes ten milliseconds, and the average for all eight falls from about forty nine to about twenty one. Round robin counts requests. Least connections counts work in progress, so new requests flow away from a busy or slow instance automatically.
Why not always pick the least loaded instance in the pool? Because in a large system, dozens of balancers each see only their own requests, or a load report that is seconds old. If they all pick the instance that looked least loaded, they all pick the same one. It's overwhelmed, and the herd moves on to the next. Michael Mitzenmacher analysed a strikingly simple alternative: choose two instances at random, and send the request to the less loaded one. Randomness prevents herding, and the comparison avoids the worst instances. It's now a standard option in proxies such as Envoy, and a good default wherever there are many balancers.
A balancer must know which instances can take traffic. A liveness check asks whether the process is stuck beyond recovery. A readiness check asks whether it should receive traffic right now. The classic mistake is calling the database from liveness checks. A brief database outage fails every check, every instance restarts at once, and the restart storm prolongs the outage. Keep liveness local and minimal.
Instances leave the pool with every deployment and scale-in. Connection draining makes that a non-event. First, the instance receives a shutdown signal. Its readiness check fails, so the load balancer stops sending it new requests. The requests already in flight finish normally. Idle keep-alive connections are closed, so clients reconnect elsewhere. Then the process exits cleanly, before its grace period runs out.
Older applications kept each user's session in one server's memory, so balancers offered sticky sessions. But load becomes uneven, and when that instance fails, baskets vanish and people are logged out. The remedy is to move session state out of the instance. A short-lived signed token can carry the user's identity with every request. A shared session store handles larger state and immediate logout, at the cost of a network call per request. And state that must survive, such as a basket the user expects tomorrow, belongs in the database. Then any instance can serve any request.
The full section also covers hashing and global load balancing with DNS and anycast. Next, in Section nine point three, we look at what happens when work arrives faster than it can be done. See you there.
What you need to know
- A load balancer spreads connections or requests over healthy instances, hides instance changes behind one address and often terminates TLS; it may sit at the edge or inside clients and sidecars.
- Layer 4 balancers forward connections by address and port, fast and protocol-blind; layer 7 balancers read each HTTP request and can route, retry and rate-limit, which per-request protocols such as gRPC over HTTP/2 require.
- Round robin ignores request cost; least connections and the power of two random choices (Mitzenmacher) adapt to load, and two random choices avoid the herding that many balancers cause when all chase the least-loaded instance.
- Liveness failures restart an instance and readiness failures only remove it from rotation; keep shared dependencies out of liveness checks so a database blip cannot restart the whole fleet.
- Connection draining stops new requests to a departing instance and lets in-flight work finish, so deployments and scale-in cause no errors.
- Sticky sessions cause uneven load and lost sessions on failure; move session state to signed tokens, a shared store or the database so any instance can serve any request.
- Global load balancing steers users to regions by DNS answers, limited by caching and TTLs, or by anycast, which announces one address from many sites and lets routing choose the nearest.
What a load balancer does
A load balancer sits between clients and a pool of interchangeable instances and decides, for each connection or request, which instance receives it. Distribution is only part of the job. A load balancer also checks which instances are healthy and stops sending traffic to those that are not; it gives clients one stable address while instances come and go with deployments and autoscaling; it often terminates TLS, so certificates are managed in one place; and at layer 7 it can route, retry, rate-limit and record every request. It is therefore the component that makes horizontal scaling invisible to clients.
Load balancing appears at several levels. A managed cloud load balancer or a pair of proxies such as HAProxy, NGINX or Envoy usually stands at the edge. Between services, balancing is increasingly done on the client side: a gRPC client library or a service-mesh sidecar proxy (Section 6.3) holds the list of healthy instances from service discovery and chooses one itself, which removes a network hop and a central component. Whatever the form, the same decisions recur: what the balancer can see, which algorithm it uses, how it decides an instance is healthy, and how instances leave without dropping work.
Layer 4 and layer 7 load balancing
The layer numbers come from the OSI network model. A layer 4 load balancer works at the transport layer: it sees IP addresses and TCP or UDP ports, picks a back end when a connection opens, and then forwards that connection's packets without looking inside them. Because it does so little, it is extremely fast and works for any protocol, including databases, message brokers and encrypted traffic it cannot read. Google's Maglev and the network load balancers of the major clouds are layer 4 designs that handle millions of packets per second.
A layer 7 load balancer is a reverse proxy that understands the application protocol, almost always HTTP. It terminates the client's connection (and usually TLS), reads each request, and opens or reuses its own connections to the back ends. That lets it route by path, host, header or cookie (/orders/* to the order pool, /menus/* to the menu pool), balance each request rather than each connection, retry a failed request on another instance, enforce rate limits, add tracing headers and log every request with its status and latency. The cost is more processing per request, a component that must hold the TLS keys, and one more place where timeouts and retries must be configured consistently with the rest of the system (Section 9.4).
One difference catches many teams out. HTTP/2 and gRPC multiplex many requests over one long-lived connection. A layer 4 balancer chooses a back end once per connection, so a client that opens one connection sends all its requests to one instance, however many instances exist: new instances added by autoscaling receive nothing until clients reconnect. Balancing gRPC traffic therefore needs a layer 7 proxy or client-side balancing per request.
A typical two-tier arrangement
Load-balancing algorithms
The algorithm decides which healthy instance gets the next connection or request. The simple ones ignore how busy instances are; the better ones use some measure of load; hash-based ones deliberately send related requests to the same place. The table summarises the common choices.
Common load-balancing algorithms
| Algorithm | How it chooses | Strengths | Weaknesses |
|---|---|---|---|
| Round robin | Each instance in turn | Simple, predictable, no state about load | Blind to request cost and to slow instances |
| Weighted round robin | In turn, in proportion to configured weights | Mixes instance sizes; ramps new versions up gradually (canaries) | Weights are static guesses; still blind to actual load |
| Random | Uniformly at random | Needs no shared state, so many balancers can use it independently | Uneven by chance, especially with few requests |
| Least connections (least outstanding requests) | Instance with the fewest requests in progress | Adapts to uneven request costs and slow instances | Needs accurate counts; each of many balancers sees only its own; new instances can be flooded |
| Latency-weighted (for example EWMA of response time) | Instance with the best recent latency, often combined with load | Steers away from degraded instances quickly | Can oscillate; a fast-failing instance looks attractive unless errors are penalised |
| Hash (source IP, header, consistent hashing) | Same key always maps to the same instance | Affinity for caches and sessions; consistent hashing moves few keys when the pool changes | Hot keys overload one instance; uneven key distribution |
| Power of two random choices | Pick two instances at random; send to the less loaded | Nearly as good as least connections, no herding, little coordination | Slightly worse than a perfect global choice with perfect information |
Worked example
Worked example: why least connections beats round robin when request costs differ
Two instances, A and B, each process one request at a time (real instances have more threads, but the effect is the same at larger scale). Eight requests arrive 10 ms apart. The first is a restaurant sales report costing 100 ms; the other seven are menu reads costing 10 ms each.
| Round robin: assignment | A gets requests 1, 3, 5, 7; B gets 2, 4, 6, 8, regardless of cost |
|---|---|
| Round robin on A | R1 runs 0 to 100 ms. R3 (arrived at 20) waits and finishes at 110: 90 ms. R5 (40) finishes at 120: 80 ms. R7 (60) finishes at 130: 70 ms |
| Round robin on B | R2, R4, R6 and R8 each finish 10 ms after arriving; B is idle half the time |
| Round robin result | Menu reads: 90, 80, 70, 10, 10, 10, 10 ms, mean 290 ÷ 7 ≈ 41 ms, worst 90 ms. All eight requests: mean (100 + 290) ÷ 8 ≈ 49 ms |
| Least connections: assignment | R1 goes to A (both idle). At 10 ms A has 1 request in progress and B has 0, so R2 goes to B; B is always free again before the next arrival, so R3 to R8 all go to B |
| Least connections result | Every menu read takes 10 ms; the report takes 100 ms. Mean (100 + 70) ÷ 8 ≈ 21 ms, worst menu read 10 ms |
| Why | Round robin counts requests, not work. Least connections counts work in progress, which by Little's law (in flight = rate × time) grows on whichever instance is slower or busier, so new requests flow away from it automatically |
The same effect appears whenever request costs vary widely (searches against lookups, exports against reads) or one instance is slower than its peers because of a noisy neighbour, a garbage-collection pause or a cold cache.
Least connections needs a meaningful count. With HTTP/2 or gRPC many requests share one connection, so balancers count outstanding requests instead; with long-lived WebSocket connections the count measures sessions, not load.
A newly started instance has zero connections and can be flooded before its caches are warm; many balancers offer a slow-start period that ramps its share up over a minute or so.
The power of two random choices
If least connections works so well, why not always pick the globally least-loaded instance? Because in a large system there is no single, current view of load. Dozens of load balancers or thousands of client-side balancers each see only their own requests, or a load report that is a few seconds old. If they all pick the instance that looked least loaded, they all pick the same one, which is promptly overwhelmed, then they all move to the next: a herd that chases the load around the pool.
Michael Mitzenmacher's doctoral thesis (1996), published as The Power of Two Choices in Randomized Load Balancing (IEEE Transactions on Parallel and Distributed Systems, 2001) and building on earlier work by Azar, Broder, Karlin and Upfal, analysed a strikingly simple alternative: choose two instances at random and send the request to the less loaded of the two. In the classic balls-into-bins model, placing n requests on n servers at random gives a most-loaded server with about log n ÷ log log n requests; with two random choices this falls to about log log n ÷ log 2, an exponential improvement, while a third choice adds only a little more. Randomness prevents herding, and the comparison avoids the worst instances.
The power of two random choices is now the default or a standard option in widely used proxies: Envoy's least-request balancer compares two randomly chosen hosts by default, and NGINX offers random two least_conn. It is a good default for client-side balancing in any system with many balancers.
Hashing and affinity
Sometimes the point is to send related requests to the same instance: to a cache node that already holds a restaurant's menu, or to the instance holding a WebSocket session. Hashing a key (restaurant identifier, user identifier, source IP) to an instance achieves this. Plain hash mod N remaps most keys whenever N changes, which empties every cache at the worst moment, during a scale-out. Consistent hashing (introduced in Section 7.4) moves only about 1 ÷ N of the keys when an instance joins or leaves. Its weakness is hot keys: the most popular restaurant at lunchtime lands on one instance. Variants such as consistent hashing with bounded loads cap each instance's share and spill the excess to the next instance on the ring.
Health checks: liveness and readiness
A load balancer must know which instances can take traffic. An active health check probes each instance periodically, for example GET /healthz every five seconds, marking it down after a few consecutive failures and up again after a few successes. Passive checks, often called outlier detection, watch real traffic and eject an instance that returns, say, five consecutive server errors. The two complement each other: active checks find instances that receive no traffic, passive checks react to failures that a trivial probe does not exercise.
Kubernetes made an important distinction explicit. A liveness probe asks whether the process is stuck beyond recovery (deadlocked, out of memory in a loop); if it fails, the instance is restarted. A readiness probe asks whether the instance should receive traffic right now; if it fails, the instance is removed from the load balancer but left running, for example while it warms its caches at start-up, or while it is shedding load. A third, the startup probe, holds the other two off while a slow application starts.
The classic mistake is to check shared dependencies in these probes. If every instance's liveness probe calls the database, a brief database outage makes every probe fail, the orchestrator restarts every instance at once, and the restart storm (cold caches, thousands of new connections) prolongs the outage long after the database is back. Readiness checks that include a shared dependency can similarly mark the whole fleet unready, turning partial errors into a total outage. Keep liveness local and minimal; let readiness reflect the instance's own ability to serve; and handle dependency failures with the patterns of Section 9.4. Load balancers guard against the extreme case by failing open: an AWS Application Load Balancer routes to all targets if all of them fail their health checks, and Envoy ignores health status when the share of healthy hosts falls below a panic threshold (50% by default), on the reasoning that sending traffic to doubtful instances is better than sending it nowhere.
Connection draining and graceful shutdown
Instances leave the pool constantly: every deployment replaces them, autoscaling removes them, and the platform moves them during maintenance. If an instance is simply killed, every request it was processing fails, and clients see errors during what should be a routine release. Connection draining (called deregistration delay on AWS) removes the instance from the balancer's rotation so that it receives no new requests, waits for the requests already in progress to finish, and only then lets it stop.
The instance must cooperate. On receiving its shutdown signal it should fail its readiness check, keep serving in-flight requests, tell clients on keep-alive connections to go elsewhere (a Connection: close header on HTTP/1.1, a GOAWAY frame on HTTP/2), finish or hand back background work, and exit before the grace period ends. In Kubernetes, removing a pod from the load balancer and sending it SIGTERM happen concurrently, so a request can still arrive just after the signal; a short pre-stop delay of a few seconds covers that race. The default grace period is 30 seconds, after which the process is killed. Long-running requests (large uploads, streaming responses, WebSockets) need either a longer grace period or a protocol by which clients reconnect elsewhere without losing work.
Graceful shutdown in five steps
Sticky sessions and their problems
Older web applications kept each user's session (who is logged in, what is in the basket) in the memory of the server that created it. Behind a load balancer this works only if all of a user's requests go back to that server, so balancers offer sticky sessions (session affinity), usually by setting a cookie that names the instance. Stickiness is a quick way to put a stateful application behind a balancer, and it has a legitimate use as a performance hint, for example to keep a user on an instance whose local cache already holds their data.
As a correctness mechanism it causes trouble. Load becomes uneven, because a few heavy users or a large office behind one IP address stay pinned to one instance whatever its load. When an instance fails, its users lose their sessions: baskets vanish and people are logged out. Scaling in and every deployment do the same unless sessions are drained, which can take as long as the longest session. New instances added by autoscaling receive only new users, so they relieve the overloaded instances slowly. And the system is no longer stateless in the sense of Section 9.1, so every one of these problems reappears in failover between zones or regions.
Externalising session state
The remedy is to move session state out of the instance, so that any instance can serve any request. There are three main options, often combined. A client-side token carries the state with each request: a signed (and if necessary encrypted) cookie or a token such as a JWT holding the user's identity and a little data. Servers need no store at all, but tokens must stay small, cannot easily be revoked before they expire, and must be protected against tampering and theft (Section 8.3). A shared session store, typically Redis or Memcached, keeps sessions in a fast external store keyed by a session identifier in the cookie; it handles larger state and immediate logout, at the cost of a network call per request and a dependency that must itself be highly available. A database suits state that must survive anything, such as a basket the user expects to find tomorrow, which is really business data rather than session data.
A practical split for the food-delivery platform: identity in a short-lived signed token, the basket in the order service's database (users switch from phone to laptop), and transient UI state in the client. No instance keeps anything a later request needs, so the load balancer can use least-request balancing with no affinity, and instances can be added, removed and replaced at will.
Global load balancing: DNS and anycast
Once a system runs in several regions, something must send each user to a suitable one: the nearest healthy region, or the region that holds their data. Global load balancing is usually done in one of two ways, often combined.
DNS-based global server load balancing uses an authoritative name server that answers the same name with different addresses depending on where the query comes from, the health of each region and configured weights: a user in Lisbon is given the address of the EU region, a user in Ohio the US region. It needs no special network, but it is coarse and slow to react. Answers are cached by resolvers for at least the record's time to live (often 30 to 300 seconds) and sometimes longer, so after a region fails some users keep arriving there for minutes. The name server sees the resolver's location rather than the user's, although the EDNS Client Subnet extension (RFC 7871) lets resolvers pass on part of the client's address.
With anycast, the same IP address is announced through the internet's routing protocol (BGP) from many sites, and each user's packets are delivered to whichever site is nearest in routing terms. Failover is a matter of withdrawing the announcement from a failed site, which takes effect quickly without waiting for DNS caches. Anycast has long been used for DNS root and resolver services; CDNs and the global load balancers of the large cloud providers use it to bring users to a nearby edge site, where layer 7 proxies terminate the connection and forward requests over the provider's network to a healthy region. Multi-region design, data placement and failover are the subject of Section 10.6.
DNS-based and anycast global load balancing
Check your understanding
Why might a newly added gRPC server behind a layer 4 load balancer receive no traffic?
Layer 4 balancing chooses a back end per connection, and gRPC clients keep one long-lived HTTP/2 connection, so existing clients never send it requests until they reconnect.
What is the difference in effect between failing a liveness probe and failing a readiness probe?
Failing liveness makes the orchestrator restart the instance; failing readiness only removes it from the load balancer until it passes again.
Why do many independent load balancers choosing the least-loaded instance perform badly?
They act on the same stale view and all pick the same instance, overloading it in turn; picking the better of two random instances avoids this herding.
Give two problems caused by keeping sessions in instance memory with sticky sessions.
Users lose their sessions when the instance fails or is replaced, and load becomes uneven because heavy users stay pinned to one instance.
Sources and further reading
Have a question about this section? ArchitectBot, the course’s AI study assistant, answers from the course first.
Ask: “We run eight instances behind a load balancer with sticky sessions and lose users' baskets on every deployment. How do we move to stateless instances safely?”