Capacity planning and performance testing

Capacity is a plan, not a hope

Capacity planning answers one question ahead of time: will we have enough of everything when demand peaks? Everything is broader than servers. It includes database connections and write throughput, cache memory, message broker partitions, cloud account quotas, third-party rate limits, licences and the people on call. Cloud elasticity makes some capacity quick to add, but not all: a database primary cannot be doubled in seconds, a payment provider may need weeks of notice to raise a limit, and autoscaling (covered in Section 9.1) reacts in minutes to a step change that arrives in seconds.

The method has four steps, repeated on a regular cycle (quarterly for steady growth, per event for launches and on-sales):

Sizing arithmetic with Little's law was worked through in Section 9.1; this section concentrates on how to obtain trustworthy numbers to put into it. The most common error in capacity planning is not bad arithmetic but a per-instance capacity measured under unrealistic conditions, and the second is forgetting a shared constraint, such as a third-party limit, that no amount of headroom in your own servers can fix.

Kinds of performance test

Performance testing is the only way to measure capacity, find bottlenecks before customers do and verify that a release has not made things worse. The tests differ in the load shape they apply and the question they answer. A load test checks the expected peak; a stress test goes beyond it to see how the system fails; a soak test runs for hours to expose slow leaks; a spike test checks reactions to sudden jumps; a breakpoint test ramps up until something breaks, to find the true limit.

Open and closed workload models

Load generators simulate users in one of two ways, and choosing the wrong one produces results that are not just inaccurate but dangerously optimistic. In a closed workload model the tool runs a fixed number of virtual users; each sends a request, waits for the response, pauses for a think time and repeats. If the system slows down, each virtual user sends fewer requests, so the load falls by itself. In an open workload model requests arrive at a set rate, independently of how quickly earlier ones are answered, as real visitors to a public website do: thousands of fans pressing Buy at 10:00 do not wait for each other.

Bianca Schroeder, Adam Wierman and Mor Harchol-Balter showed (Open Versus Closed: A Cautionary Tale, NSDI 2006) that the two models can give very different response times for the same system and the same average load, and that scheduling or capacity decisions tuned under one can be wrong under the other. Internal systems with a fixed population of clients, such as a call-centre application with 300 agents, are reasonably closed. Public internet traffic, event-driven consumers fed by a queue and on-sales are open, or close to it. For such systems an open-model test is the realistic one; k6's constant-arrival-rate and ramping-arrival-rate executors, for example, implement it, while its constant-vus executor is closed.

Coordinated omission

Gil Tene, of Azul Systems and the author of the HdrHistogram library, named a measurement error that affects many latency tests and monitoring tools: coordinated omission. A closed-loop load generator waits for each response before sending the next request. When the system stalls, the generator stalls with it, so the requests that real users would have sent during the stall, and that would all have been slow, are never sent and never measured. The tester coordinates with the system it is testing and omits exactly the bad samples.

Consider a test that intends to send one request every 10 ms for 100 seconds against a service that normally answers in 1 ms. At 50 seconds the service freezes for 10 seconds (a long garbage-collection pause, say). The closed-loop tester records about 9,000 fast samples and one 10-second sample, so it reports a p99 of 1 ms and even a p99.9 of 1 ms. On the intended schedule, however, about 1,000 requests would have arrived during the freeze, waiting between 10 seconds and 10 ms, about 5 seconds on average. Ten per cent of all requests were badly delayed: the true p95 is about 5 seconds and the true p99 about 9 seconds.

The remedies are to use open-model (arrival-rate) generators for open systems, to measure latency from the time a request was intended to be sent rather than when it was actually sent, to record full histograms rather than averages, and to be suspicious of any latency report whose maximum is far above its p99.9 with few samples in between. Tene's modified HTTP benchmarking tool wrk2 and HdrHistogram's correction options exist for exactly this. The same error appears in production monitoring when a client library measures only the requests it managed to send.

Tools

Four open-source tools cover most needs. Apache JMeter is the veteran: Java-based, with a graphical test-plan editor, many protocol plug-ins and thread groups that are closed by default. Gatling writes tests as code (Scala, Java and Kotlin, with JavaScript and TypeScript added more recently), is efficient at high load and has open and closed injection profiles. k6, maintained by Grafana Labs, runs JavaScript test scripts on an engine written in Go, has explicit open- and closed-model executors and pass-or-fail thresholds suited to continuous integration. Locust describes user behaviour in Python and distributes load across worker processes; its users are closed-loop. Commercial and cloud-hosted services add large-scale distributed load generation and reporting.

The tool matters less than the scenario. A good script reproduces the real mix of journeys (from production analytics: perhaps 70% browsing, 20% basket changes, 10% checkout), realistic think times, realistic data (different users, restaurants and seats, not the same one 10,000 times) and the correct workload model, and it checks responses for correctness, because a fast error page is not a successful request.

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    onsale: {
      executor: 'ramping-arrival-rate',   // open model: arrivals per second
      startRate: 100, timeUnit: '1s',
      preAllocatedVUs: 2000, maxVUs: 20000,
      stages: [
        { target: 4900, duration: '2m' },  // ramp to 1.5x design load
        { target: 4900, duration: '20m' }, // hold
      ],
    },
  },
  thresholds: {                            // fail the run if missed
    http_req_failed: ['rate<0.005'],
    http_req_duration: ['p(99)<500'],
  },
};

export default function () {
  const res = http.get('https://perf.tickets.example/api/events/42/seats');
  check(res, { 'status is 200': (r) => r.status === 200 });
}

Realistic environments and data

A performance test is only as trustworthy as its environment. A test database with 10,000 rows whose indexes fit in memory says nothing about production with 200 million rows that do not. The checklist below catches the usual differences.

Size and shape of data: production-scale volumes and realistic distributions, including hot keys such as the most popular restaurants or the best seats, which create contention a uniform test never sees. Same building blocks: the same instance types, container limits, database configuration, connection pool sizes and network topology (including cross-zone hops). Cache state: test both warm and cold caches, because a deployment or failover empties them (Section 7.6). Dependencies: third-party services replaced by stubs that reproduce their real latency distribution and rate limits, not instant responses. Isolation: no other team's tests sharing the environment, or results are noise. The generator itself: enough load-generator capacity in the right locations; a generator running at 100% CPU measures itself.

Some organisations test in production, carefully, because no copy is ever quite like it: synthetic users and test events that are excluded from business reporting, traffic shadowing (copying real requests to a new version and discarding its responses), and gradual ramps with automatic abort conditions. This needs the same safeguards as chaos experiments: a small blast radius, a kill switch and close watching of SLOs.

Performance budgets

A performance budget is a limit that a change may not exceed, agreed in advance and checked automatically, so that performance does not erode one small regression at a time. For web front ends, budgets cover page weight, script size and the Core Web Vitals, for which Google's thresholds for a good experience, measured at the 75th percentile of page loads, are a Largest Contentful Paint of 2.5 seconds or less, an Interaction to Next Paint of 200 ms or less and a Cumulative Layout Shift of 0.1 or less.

For back ends, a latency budget divides an end-to-end SLO between the hops of a journey. The food-delivery checkout has a p99 target of 800 ms; the team allocates 20 ms to the gateway, 60 ms to the order service, 80 ms to pricing, 450 ms to payment (most of it the external card provider), 40 ms to the database, 100 ms to network and serialisation, and keeps 50 ms in reserve. Each team now knows its own target, and a change that pushes pricing to 150 ms is visible as a budget breach before it becomes an SLO breach. Strictly, percentiles of the parts do not add up to the percentile of the whole, so treat the split as an engineering allocation and verify the end-to-end figure separately.

Budgets are enforced by running a short, stable performance test in the delivery pipeline and failing the build when a threshold is crossed, as the k6 thresholds above do. This is a fitness function for performance. Pipeline tests must be repeatable more than realistic: same environment, same data, same load, so that a 10% change means a real regression and not noise.