Performance, Scalability, Resilience and Operability

Welcome to Chapter 9, Performance, Scalability, Resilience and Operability. A design that meets its targets in a diagram can still fall over at half past twelve on a Friday, when every office in the city orders lunch. The difference is rarely clever code. It is a set of architectural decisions about how capacity is added, how load is spread, how work waits, how failures are contained, and how the people running the system know what it is doing.

This chapter is for developers and architects who know the distributed styles of Chapter 6, and who must keep a system fast and running under real load. It builds on Chapter 2, which made performance and availability precise as testable quality attribute scenarios. Here you learn to achieve those scenarios in a running system, and keep them true on the busiest day of the year. It links back to Chapter 7 for replication, partitioning and caching, rather than repeating them. And it leads to Chapter 10, on the cloud platforms, Kubernetes features and multi-region designs these systems usually run on.

Two running examples carry through the chapter. A food delivery platform, with sharp lunchtime and evening peaks. And a ticket on-sale, where demand rises from almost nothing to many times the system's capacity within seconds. Section nine point one compares scaling up with scaling out, uses the scale cube to separate cloning, splitting by function and splitting by data, and explains why autoscaling on CPU is often the wrong choice. Section nine point two covers what sits in front of the instances: load balancers, the algorithms they use, health checks, connection draining, and why sticky sessions should give way to session state kept outside the instance. Section nine point three is about what happens when work arrives faster than it can be done: bounded queues, back pressure, load shedding and rate limiting, which decide what is accepted, what waits and what is refused politely. Section nine point four deals with failure between services: timeouts and deadlines, retries with backoff and jitter, circuit breakers, bulkheads and fallbacks.

Section nine point four's worked example shows how one missing timeout lets a single slow dependency take down everything that calls it. The payment provider normally answers in zero point three seconds. Today it has stopped answering. The order service calls it without a timeout, so one by one, all two hundred of its threads end up waiting on payment calls. Now checkout fails. But so do order history and tracking, which never touch payments, because there are no free threads left to serve them. The database is perfectly healthy, but no thread is free to call it. Timeouts, circuit breakers and bulkheads are the patterns that stop this kind of cascade.

The second half turns from mechanisms to operation. Section nine point five shows how to state reliability from the user's point of view, and how an error budget turns an availability target into an amount of allowed failure that teams can spend on change. Section nine point six covers the logs, metrics and traces that let engineers ask new questions of a production system, and distributed tracing with OpenTelemetry. Section nine point seven closes the loop between forecasts and evidence, with load, stress, soak, spike and breakpoint tests, and a capacity plan for a ticket on-sale. And Section nine point eight deals with the failures that happen anyway: deliberate experiments to find weaknesses before customers do, clear roles during incidents, and blameless reviews that change the system rather than the people.

This diagram from Section nine point five puts three terms on one scale, using the availability of checkout over a rolling twenty eight days. The service level objective, or SLO, is the internal target: here, ninety nine point nine per cent. The service level agreement, or SLA, is a contract with partners, set looser, at ninety nine point five per cent. The service level indicator, or SLI, is the measurement. This month it is ninety nine point nine four per cent, inside the zone where the service is within its budget. Below the SLO, the target is missed, and a written error budget policy says what happens next. Only below the SLA are service credits owed. The gap between the SLO and the SLA is the warning zone. The error budget is simply one minus the SLO. Over thirty days, a target of ninety nine point nine per cent allows forty three point two minutes of complete outage, or one request in every thousand failing.

The sections form a chain, and they lead to one idea. Every resource is finite, so every system has a breaking point. The architect's job is to decide in advance what happens when it is reached. A well designed system under excessive load slows down gracefully, and sheds the least important work first. It refuses clearly, rather than timing out silently. It isolates the failing part. And it recovers by itself when the pressure passes. A poorly designed one queues without limit, retries until it collapses, and takes its neighbours with it. These mechanisms are cheap to add at design time and expensive to retrofit. Whether a service is stateless, whether a queue is bounded and whether a request carries a deadline are all decided long before the first incident.

By the end of the chapter, you will be able to choose how to scale, and size a fleet for a known peak with Little's law and headroom. You will be able to combine timeouts, deadlines, retries with backoff and jitter, circuit breakers, bulkheads and fallbacks to contain failures. You will be able to define service level indicators and objectives from the user's point of view, calculate error budgets, and design alerts on how fast they are being spent. And you will be able to plan capacity and performance tests, run chaos experiments safely, and lead blameless post-incident reviews.

Because the sections form a chain, reading them in order works best. But each one also stands on its own, so you can go straight to the section you need. Allow about three and a half hours to read the whole chapter. At the end of each section there are self-check questions. Answer them in your head before you look at the answers. When you are ready, there is an optional chapter test, which takes about forty five minutes. It has twenty multiple choice questions and two written answers. The pass mark is eighty per cent.

That is the shape of the chapter. Start with Section nine point one, on scaling up, scaling out and the scale cube. See you there.