Performance and scalability: latency, throughput, percentiles and load

Two different questions: how fast, and how much

Performance is the quality people mention first and specify worst. It covers two different questions. Latency asks how long one piece of work takes: how many milliseconds between tapping 'Pay' and seeing 'Payment accepted'. Throughput asks how much work the system completes per unit of time: 1,200 payments per second, 40,000 orders per hour. The two are related but not interchangeable. A coach carrying 50 people from London to Oxford in 100 minutes has worse latency and much better throughput than a sports car carrying two people in 60 minutes.

In software the same tension appears everywhere. Batching database writes improves throughput but makes each individual write wait for its batch. Adding a cache improves both for repeated reads but does nothing for the first. A good performance requirement therefore says which question it is about, and usually states both: 'at 1,200 payments per second (throughput), 99% complete within 800 ms (latency)'. ISO/IEC 25010 files these under performance efficiency as time behaviour, resource utilisation and capacity (Section 2.2).

Response time = service time + waiting time

When you measure latency from the client's side you get the response time: everything that happened between sending the request and receiving the answer. It is useful to split it into parts. The service time is the time a resource actually spends working on your request: parsing it, running the query, rendering the result. The waiting time is the time your request spends in a queue because the resource is busy with someone else's. Network transfer time is added on top, in both directions.

The distinction matters because the two parts respond to different remedies. Service time falls when you make the work itself cheaper: a better index, a smaller payload, a faster algorithm. Waiting time falls when you add capacity or reduce load, and it is often much larger than the service time when a system is busy. Many 'slow database' investigations end with the discovery that each query takes 5 ms but waits 300 ms for a free connection in the pool.

Percentiles: why averages mislead

Response times are never a single number; they form a distribution, and it is almost always skewed. Most requests are quick, and a minority are slow because they hit a cold cache, a garbage-collection pause, a lock, a retry or an overloaded neighbour. The arithmetic mean blends the two groups into a figure that describes nobody. Professionals use percentiles instead. The 50th percentile (p50, the median) is the value half of requests beat; p95 is the value 95% of requests beat; p99 and p99.9 describe the slowest 1% and 0.1%.

The slow end of the distribution is called the tail, and it matters more than it seems. The users in the tail are often your best customers: the ones with the biggest baskets, the longest histories and the most data. A user who loads 40 resources to view one page experiences the slowest of those 40, not the average. And the chance that a 40-resource page hits at least one p99 request is 1 − 0.99⁴⁰ ≈ 33%: a 'rare' 1% event affects a third of page views.

# Why the mean misleads: 1,000 response times from one minute of traffic
import statistics

times_ms = [80] * 900 + [150] * 80 + [1200] * 15 + [4000] * 5

mean = statistics.mean(times_ms)
cuts = statistics.quantiles(times_ms, n=1000, method="inclusive")
p50, p95, p99, p999 = cuts[499], cuts[949], cuts[989], cuts[998]
print(f"mean {mean:.0f} ms, p50 {p50:.0f}, p95 {p95:.0f}, p99 {p99:.0f}, p99.9 {p999:.0f}")
# mean 122 ms, p50 80, p95 150, p99 1200, p99.9 4000

Tail latency at scale

The tail becomes the main event in large systems that use fan-out, where one user request is split into many parallel requests to other servers and the answer must wait for all of them. A search engine, a social feed or a recommendations page may consult tens or hundreds of back-end shards for every query. Jeffrey Dean and Luiz André Barroso of Google analysed this in The Tail at Scale (Communications of the ACM, 2013).

Their illustration is simple. Suppose each server answers in 10 ms typically but takes 1 second for one request in 100. A user request that waits for one server is slow 1% of the time. A user request that waits for 100 such servers is slow whenever any one of them is slow: 1 − 0.99¹⁰⁰ ≈ 63% of the time. A rare event at one server becomes the common case for the user. Dean and Barroso report measurements from a Google service in which the 99th-percentile latency for a single leaf request was 10 ms, but the 99th percentile for waiting on all leaf requests was 140 ms.

Because you cannot eliminate every source of variability (shared hardware, background jobs, garbage collection, network queues), they argue for tail-tolerant designs, much as fault-tolerant designs accept that hardware fails. Their techniques include hedged requests (send a second copy to another replica if the first has not answered within, say, the 95th-percentile time, and use whichever returns first), tied requests that cancel each other, finer-grained partitions that can be rebalanced, temporarily excluding slow machines, and returning a good-enough result when most but not all shards have answered. In one benchmark they describe, hedging after 10 ms cut the 99.9th-percentile latency from 1,800 ms to 74 ms while sending only about 2% more requests.

Little's law: how many requests are in flight

Little's law, proved by John Little in 1961, is the most useful single formula in capacity planning. For any stable system, the average number of items inside it (L) equals the average arrival rate (λ, lambda) multiplied by the average time each item spends inside (W): L = λ × W. It holds regardless of the arrival pattern or the order of service, provided the system is stable (on average, as many items leave as arrive) and you use consistent units.

For software, L is the number of requests in progress at once (the concurrency), λ is the request rate and W is the response time. If a service receives 200 requests per second and each spends 0.25 seconds inside, then on average 200 × 0.25 = 50 requests are in progress at any moment. That number drives thread pools, connection pools, memory and licence counts. The law also works backwards: if you can hold at most 50 requests at once and each takes 0.25 s, you cannot sustain more than 50 ÷ 0.25 = 200 requests per second.

Utilisation and queueing: why systems fall off a cliff

Utilisation (ρ, rho) is the fraction of time a resource is busy. A server that receives 40 requests per second, each needing 20 ms of service, is busy 40 × 0.020 = 0.8 of the time: 80% utilised. Intuition suggests that a resource 80% busy should be only a little slower than one 50% busy. Queueing theory says otherwise, because requests do not arrive evenly; they bunch together, and when they bunch, the later ones wait.

The simplest model is the M/M/1 queue: random arrivals, random service times and a single server. Its mean response time is W = S ÷ (1 − ρ), where S is the mean service time. At 50% utilisation, W = 2S. At 80%, W = 5S. At 90%, W = 10S. At 99%, W = 100S. The denominator shrinks towards zero as utilisation approaches 100%, so response time grows without limit. Real systems are not exactly M/M/1, and multi-server pools behave better at the same utilisation, but the shape is universal: flat, then a knee, then a cliff.

The practical lessons are direct. Do not plan to run critical resources near 100% busy; leave headroom so that bursts do not push you over the knee. Watch utilisation and queue length together (Brendan Gregg's USE method checks utilisation, saturation and errors for every resource). And remember that the cliff makes small changes dangerous: at 90% utilisation, a rise in traffic of only about 6% takes you to 95% and doubles the mean response time. Chapter 9 covers queues, back pressure and capacity planning in Asynchrony, queues and back pressure.

Scalability: load, elasticity, scaling up and out

Scalability is the ability to handle more load by adding resources, ideally with capacity growing in proportion and without redesign. It only means something once you say what 'load' is. Load parameters differ between systems: requests per second for an API, concurrent connections for a chat service, the ratio of reads to writes for a social network, data volume for an analytics platform, or the number of followers of the most-followed account for a feed. A scalability scenario names the load parameter, the growth, and the response that must be preserved: 'when daily orders grow from 50,000 to 500,000, p95 checkout latency stays below 1.5 s by adding instances, with cost per order no more than 20% higher'.

Elasticity adds time: the ability to add and remove capacity automatically as load rises and falls, so that a ticketing site can grow for a sale and shrink afterwards. It is a large part of the economic case for cloud platforms. There are two basic ways to add capacity. Vertical scaling (scaling up) gives one machine more CPU, memory or faster storage; it is simple, needs no change to the software, and runs into hard limits and rising prices. Horizontal scaling (scaling out) adds more machines and spreads the work; it has no fixed ceiling but requires the software to be designed for it, with stateless instances, load balancing and partitioned data. Chapter 9 covers the strategies in detail in Scaling strategies: vertical, horizontal and the scale cube.

Amdahl's law: the limit of parallelism

Adding processors or nodes only helps with work that can be done in parallel. Amdahl's law, from Gene Amdahl's 1967 paper on the limits of parallel computing, captures this. If a fraction s of the work is serial (it must be done one step at a time) and the rest, 1 − s, can be split perfectly across N processors, the best possible speed-up is speed-up(N) = 1 ÷ (s + (1 − s) ÷ N). As N grows without limit, the speed-up approaches 1 ÷ s. With 10% serial work, no number of processors can make the job more than ten times faster.

The Universal Scalability Law

Amdahl's law assumes that extra processors never make things worse; the speed-up just levels off. Real distributed systems often do get worse: past some point, adding nodes reduces throughput. Neil Gunther's Universal Scalability Law (USL) models this with two penalties. Contention (α, alpha) is the cost of queueing for shared resources, the same effect as Amdahl's serial fraction. Coherency (β, beta) is the cost of keeping data consistent between nodes: every node must exchange information with every other, so this cost grows roughly with the square of the number of nodes.

The relative capacity is C(N) = N ÷ (1 + α(N − 1) + βN(N − 1)). With β = 0 it reduces to Amdahl's law. With any β above zero, capacity reaches a peak at about N = √((1 − α) ÷ β) and then falls. For α = 0.03 and β = 0.002, the peak is near √(0.97 ÷ 0.002) ≈ 22 nodes, at a capacity of about 8.6 times one node; at 32 nodes capacity has dropped back to about 8.2. Gunther describes the model in Guerrilla Capacity Planning (2007). Its practical value is that α and β can be fitted to a handful of load-test measurements, which lets you predict where scaling will stop paying off before you buy the hardware.

Tactics for performance: an overview

Bass, Clements and Kazman group performance tactics into two families. The first is to control resource demand: reduce how much work arrives or how much each piece costs. The second is to manage resources: make the available resources work harder or add more of them. Most real designs combine several tactics; each also has costs elsewhere, which is why they are architectural decisions.