Scaling strategies: vertical, horizontal and the scale cube

What scaling means in practice

Section 2.4 defined scalability as the ability to handle more load by adding resources, and showed with Amdahl's law and the Universal Scalability Law why adding resources does not always add capacity. This section is about the practical choices: which resources to add, where, when and how automatically. The starting point is always the same question: what is the load parameter, and which resource runs out first? A food-delivery platform's order API at lunchtime is limited by requests per second; its dispatch service by the number of active couriers whose positions must be matched every few seconds; its database by write transactions per second and by connections. Each needs a different answer.

Scaling a system really means scaling its current bottleneck, then finding the next one. Doubling the web tier of a system whose database is 95% busy buys nothing except a larger bill and a database that is now pushed harder. So every scaling decision starts with measurement: utilisation and saturation for each resource (Brendan Gregg's USE method, covered in Observability), and a load test that finds where latency starts to climb (Capacity planning and performance testing).

Vertical scaling: a bigger machine

Vertical scaling (scaling up) means giving the existing machine more CPU cores, memory, faster disks or a faster network. Its great virtue is that nothing in the software changes. A single relational database on a larger server keeps its transactions, its joins and its simple consistency; an application keeps its in-memory caches and its local files. In the cloud, resizing is an API call followed by a restart. The largest cloud virtual machines now offer hundreds of virtual CPUs and terabytes of memory, which is far more than most business systems will ever need, and some very busy websites have publicly described serving enormous traffic from a small number of powerful database servers.

The limits are real, though. There is a largest machine, and the price per unit of capacity rises steeply near the top of the range. One machine is one failure domain: when it fails, or is restarted to be resized or patched, everything on it stops, so scaling up alone does nothing for availability. Software often cannot use a bigger machine fully: a single-threaded process, a global lock or a hot database row limits the speed-up exactly as Amdahl's law predicts. And capacity comes in large, slow steps, so you pay for the peak around the clock. Vertical scaling is nevertheless the right first move far more often than fashion suggests, especially for databases, where scaling out is expensive in design effort.

Horizontal scaling: more machines

Horizontal scaling (scaling out) means running more instances and spreading the work between them, usually with a load balancer in front. It scales much further, in small and cheap steps, and the same redundancy that adds capacity also tolerates failure: with eight instances, losing one costs an eighth of capacity rather than all of it. Cloud platforms make adding an instance a matter of seconds or minutes, which is the basis of autoscaling.

The price is that the software must be designed for it. Any instance must be able to handle any request, which means instances must not keep anything that a later request depends on (the next section). Work must be divisible: a job that needs one global lock does not run faster on ten machines. Data must be reachable from every instance, which moves the pressure to the shared data tier. And the system becomes distributed, with the partial failures, timeouts and retries that Section 6.1 describes. Most real systems combine the two: stateless application tiers scale out, while databases scale up for as long as they reasonably can and then scale out with replicas and partitions.

The scale cube: three different ways to scale out

Martin Abbott and Michael Fisher, who ran technology at eBay and PayPal, describe scaling out with the scale cube in The Art of Scalability (first edition 2009, second edition 2015). It separates three independent ways of splitting a system, each drawn as one axis of a cube whose origin is a single instance doing everything.

The X axis is horizontal duplication, or cloning: run N identical copies of the whole application or service and spread requests between them. It is the simplest and cheapest axis, and it works well for stateless request handling and read traffic. It does nothing for data size or write volume, because every clone still talks to the same data, and it does nothing for the size of the codebase or the number of teams working on it.

The Y axis splits the system by function: different services for menus, orders, dispatch and payments, each running only the code and data for its capability. Each service can then scale, fail and be deployed separately, and each is smaller and easier to understand. This is the axis behind microservices (Section 6.3), with all their costs. The Z axis splits by data: every copy runs the same code, but each is responsible for only a slice of the customers, tenants, regions or records, chosen by a key such as city or customer identifier. Database sharding is the Z axis applied to storage (Section 7.4); a cell-based architecture applies it to whole stacks, so that a fault or a bad deployment in one cell affects only that cell's customers.

The axes combine. The platform's order service (a Y-axis split) runs as many clones (X) in each regional cell (Z). Abbott and Fisher's practical advice is to use the cheapest axis that solves the actual problem: X for request volume, Y when parts of the system have genuinely different scaling, failure or team needs, and Z when one data set or one blast radius has become too large. Choosing Z because it sounds impressive brings routing, rebalancing and cross-partition queries long before they are needed.

Statelessness: the precondition for cloning

X-axis scaling works only if any clone can serve any request. A stateless service keeps nothing between requests in its own memory or on its own disk; everything a later request needs lives in a shared store or travels with the request. The state that typically hides in an instance is easy to list once you look: user sessions held in memory, uploaded files written to local disk, in-memory caches that are treated as the source of truth, scheduled jobs that must run on exactly one machine, in-process locks guarding a shared resource, and counters for rate limits.

Each has a standard remedy: sessions go to a shared store or a signed token (Section 9.2); files go to object storage; caches become a performance optimisation that can be lost at any time, or move to a shared cache (Section 7.6); scheduled jobs use a scheduler with leader election or a queue; locks move to the database or a coordination service; rate-limit counters move to a shared store. The twelve-factor app's rule that processes are stateless and share nothing (Section 10.3) is this same idea. Statelessness is cheap to design in and expensive to retrofit, because state leaks into code in many small places.

Autoscaling: reactive, scheduled and predictive

Autoscaling closes a feedback loop: measure load, compare it with a target, change the number of instances. Reactive autoscaling does this continuously from a metric. Target tracking is the common form: keep average CPU at 60%, or in-flight requests per instance at 30. The Kubernetes Horizontal Pod Autoscaler computes desired = ceil(current replicas × current metric ÷ target): six pods averaging 45 in-flight requests against a target of 30 become ceil(6 × 45 ÷ 30) = 9. By default it re-evaluates every 15 seconds and ignores deviations within 10% of the target.

Scheduled autoscaling changes capacity at known times: raise the minimum to 18 instances at 11:15 every weekday before the lunchtime peak, lower it at 14:00. It is crude and very effective for predictable patterns, and it is the only way to have capacity in place before a step change such as a ticket on-sale at 10:00 sharp. Predictive autoscaling, offered by the major cloud providers, forecasts load from history (daily and weekly cycles) and scales ahead of it, with reactive scaling as a safety net for surprises.

The crucial property of any reactive scheme is lag. Metrics are collected and averaged over tens of seconds; the autoscaler acts on its next evaluation; a new virtual machine takes a minute or more to boot, a container image must be pulled, a JVM or Python service must start, warm its caches and connection pools and pass its readiness check before the load balancer sends it traffic. Under a cluster, the nodes themselves may have to be added first by a second, slower autoscaler. From the moment load rises to the moment new capacity is useful is commonly two to ten minutes. Reactive autoscaling therefore handles gradual change; sudden change must be absorbed by headroom, by queues (Section 9.3) or by capacity scheduled in advance.

Which metric to scale on, and why CPU is often wrong

CPU utilisation is the default signal in most platforms because it is always available. It is the right signal only when the service is CPU-bound: image resizing, encryption, heavy serialisation. Most business services are not. An order API spends most of each request waiting for the database, the payment provider and other services; its CPU may sit at 25% while every worker thread is busy waiting and new requests queue. Scaling on CPU then does nothing until it is far too late. The opposite failure also happens: a garbage-collection storm or a JIT warm-up raises CPU without any rise in load, and the autoscaler adds instances that are not needed.

A better signal is one that is directly tied to the resource that runs out. For request-response services that is usually concurrency: the number of requests in flight per instance, compared with the number of worker threads or connections it has. Little's law makes the link exact (in flight = arrival rate × response time), and it reacts both to more traffic and to slower responses. For queue workers it is the backlog per worker or the age of the oldest message, which expresses how long the queue will take to drain. Latency is tempting but dangerous as a scaling signal: it rises for reasons more instances cannot fix, such as a slow database, and scaling out a service whose dependency is struggling sends that dependency even more load. Use latency for alerting, and scale on a measure of work.

Cooldowns, stabilisation and flapping

An autoscaler that reacts to every wobble will flap: add instances when a burst arrives, remove them a minute later when the burst passes, then add them again, paying the start-up cost each time and disturbing caches and connection pools. It can also chase its own tail: new instances are cold and slow, so latency or CPU per request rises briefly after scaling out, which looks like more load.

The standard defences are asymmetric. Scale out quickly, because under-capacity hurts users; scale in slowly, because over-capacity only costs money. A cooldown or stabilisation window makes the autoscaler wait after an action, or take the highest recommendation over a recent window before scaling in: the Kubernetes autoscaler, by default, uses a five-minute (300-second) window for scaling down and none for scaling up. A tolerance band ignores small deviations. Minimum and maximum instance counts bound the loop: the minimum keeps enough capacity to survive the loss of a zone and the first minutes of a surge, and the maximum protects the database, downstream services and the budget from an autoscaler that misreads a problem as load. Scaling in must also be graceful: an instance chosen for removal must stop receiving new requests and finish the ones in progress (connection draining, Section 9.2).

Scaling the data tier

Stateless application tiers are easy to scale; that is the point of making them stateless. The pressure then lands on the databases, caches and brokers they share, which is why the data tier is usually the real limit of a system. The tools were covered in Chapter 7 and are summarised here in the order professionals usually reach for them. First, reduce the work: indexes, better queries, fewer round trips per request. Second, absorb reads with caching (Section 7.6) and read replicas, accepting replication lag where the use case allows. Third, protect connections: each application instance holds a pool, so the total grows with the fleet, and a connection pooler such as PgBouncer lets thousands of client connections share a few hundred database connections (PostgreSQL's default max_connections is only 100). Fourth, scale up the primary, which is often cheaper than any redesign. Only then partition writes along the Z axis by a key such as city or customer (Section 7.4), or move to a distributed SQL database.

Two warnings apply. First, autoscaling the application tier can harm the data tier: every new instance opens connections and warms its cache from the database, so a large scale-out event can itself overload the database. Second, some limits are logical, not physical: a single hot row (the stock count for the last 500 tickets of a concert, or a global sequence number) serialises writes however large the server is. Those limits are removed by design, for example by splitting the stock into several counters or by reserving seats in batches, not by hardware.