Timeouts, retries, circuit breakers, bulkheads and graceful degradation

Failure is normal; spreading it is optional

In a distributed system some dependency is always slow or failing: a payment provider has a bad minute, a database fails over, a network link drops packets, a deployment goes wrong. Section 6.1 showed why these partial failures cannot be prevented and why a caller often cannot even tell what happened. The architectural question is therefore not how to avoid failure but how to stop a local failure becoming a global one. A cascading failure is the typical way systems die: a slow dependency makes its callers slow, their resources fill with waiting requests, they in turn become slow for their own callers, and within seconds a problem in one minor component has taken out the whole platform.

Michael Nygard catalogued the patterns that interrupt such chains in Release It! (2007; second edition 2018), drawing on outages he had investigated. His central observation is that integration points, every place where one system calls another, are the main source of instability, and that each one needs deliberate protection. The patterns in this section (timeouts, deadlines, retries with backoff, circuit breakers, bulkheads, fallbacks, idempotency and hedging) are the standard toolkit. They are tactics in the sense of Section 2.5: each is a single decision aimed at availability, and each has a cost.

Timeouts: the first and most important pattern

A timeout bounds how long a caller waits. Without one, a call to a dependency that never answers holds a thread, a connection and memory for as long as the operating system keeps the connection open, which can be many minutes. Many client libraries have no timeout by default (Python's widely used requests library waits indefinitely unless told otherwise), so the absence of a timeout is a design decision made by omission. The worked example below shows the consequence.

Two timeouts usually matter. The connect timeout bounds establishing a connection and can be short, a few hundred milliseconds inside a data centre, because a healthy server accepts connections almost immediately. The request timeout bounds the whole exchange and should come from measurement: slightly above the dependency's high percentile latency (p99 or p99.9) under normal load, so that it cuts off pathological waits without failing ordinary slow requests. It must also fit inside the caller's own budget. If the gateway gives up after 2 seconds but the order service waits 10 seconds for pricing, the order service keeps working for 8 seconds on a request nobody is waiting for. Timeouts should therefore shrink as you go down a call chain, which is the idea behind deadlines.

Deadlines and their propagation

A deadline is an absolute point in time by which the whole request must be finished. Deadline propagation passes the remaining time to every downstream call, so that each service knows how much budget is left rather than applying a fixed timeout of its own. gRPC builds this in: a client sets a deadline, the remaining time travels with the call, and a server can check whether the deadline has passed and stop work. Over HTTP, teams propagate a header with the remaining milliseconds or the absolute deadline, ideally through the same middleware that propagates tracing context (Section 9.6).

Propagation gives two benefits. A service can fail fast when the time left is obviously too short for the work, rather than starting a database query that cannot finish in time. And when the user gives up, the work below stops too, instead of continuing to consume capacity during exactly the overload that caused the delay. This is load shedding of dead requests, one of the most effective protections a system under stress can have.

Retries with exponential backoff and jitter

Many failures are transient: a dropped packet, a connection reset during a failover, a brief 503 from an overloaded instance. Retrying turns these into successes the user never sees. But retries are also load, and badly designed retries are among the most common causes of outages. Four rules make them safe. Retry only transient errors (timeouts, connection failures, 502, 503, 504, and 429 after the indicated delay), never errors that will recur, such as 400 or 404. Retry only idempotent operations or those carrying an idempotency key, because a timed-out request may have succeeded. Retry a small number of times, typically two or three. And wait between attempts, with exponential backoff: a delay that doubles with each attempt, up to a cap.

Backoff alone is not enough. If a thousand clients fail at the same moment, as they do when a dependency blips, they all retry after 100 ms, then all after 200 ms, arriving in synchronised waves that can keep the dependency down. Marc Brooker's article Exponential Backoff And Jitter (AWS Architecture Blog, 2015) simulated this and showed that adding randomness, jitter, greatly reduces both the total work and the time to complete. His recommended full jitter sleeps for a random time between zero and the exponential bound: sleep = random(0, min(cap, base × 2attempt)). Jitter belongs on anything many clients do at once: retries, reconnections, cache expiry times and scheduled jobs.

import random, time

class TransientError(Exception):
    """Timeouts, connection resets, 502, 503 and 504 responses."""

def call_with_retries(fn, attempts=3, base=0.1, cap=2.0):
    for attempt in range(attempts):
        try:
            return fn()
        except TransientError:
            if attempt == attempts - 1:
                raise                      # give up: let the caller decide
            ceiling = min(cap, base * 2 ** attempt)
            time.sleep(random.uniform(0, ceiling))   # full jitter

Retry storms and retry budgets

Retries multiply across layers. Suppose a tap in the mobile app passes through the API gateway, the order service and the payment client, and each of the three makes up to three attempts. If the payment provider is down, one user action produces 3 × 3 × 3 = 27 calls to it, and an outage that should have meant a steady error rate becomes a retry storm of 27 times the normal load, which keeps the provider down even after the original cause is fixed. Users tapping again add a fourth layer.

Three defences work together. First, retry at one layer only, usually the one closest to the failing dependency, and have other layers pass the error up; Google's SRE book recommends that a server which is overloaded say so explicitly, so that callers above do not retry. Second, use a retry budget: the SRE book describes allowing retries only while they are below 10% of a client's requests, which caps the extra load from retries at about 1.1 times normal instead of up to three times. AWS SDKs implement a similar quota of retry tokens, and gRPC's retry policy includes throttling for the same purpose. Third, combine retries with a circuit breaker, so that a dependency known to be down is not retried at all.

Circuit breakers

A circuit breaker, named by Nygard after the electrical device, wraps calls to a dependency and watches their outcomes. While the dependency is healthy, the breaker is closed and calls pass through, while failures and timeouts are counted. When failures cross a threshold, for example half of the last 20 calls, the breaker opens: for a set period every call fails immediately without touching the network, which protects the caller's threads and gives the dependency room to recover. When the period expires the breaker becomes half-open and lets a few trial calls through. If they succeed it closes; if one fails it opens again for another period. Martin Fowler's 2014 description made the pattern widely known, and Netflix's Hystrix library (now in maintenance mode) popularised it in microservices; Resilience4j for Java and Polly for .NET are the common libraries today, and service meshes offer related protections such as concurrency limits and outlier detection (in Envoy these are called circuit breaking, although they work differently).

The settings deserve thought. What counts as a failure? Timeouts and 5xx responses should; 4xx responses usually should not, because they reflect the caller's request rather than the dependency's health. Slow successes can also count, so that a dependency that answers in 5 seconds is treated as failing. The window must be large enough to avoid tripping on a couple of unlucky calls (libraries require a minimum number of calls) and the open period long enough for recovery but short enough not to prolong an outage. Breakers should be per dependency, and sometimes per endpoint, so that a failing reporting endpoint does not cut off ordering. Retries should treat an open breaker as a final answer rather than something to retry. And every change of state should be logged and graphed, because an open breaker is a symptom someone needs to see.

import time

class CircuitOpen(Exception):
    pass

class CircuitBreaker:
    def __init__(self, failure_threshold=5, reset_after=30.0):
        self.failure_threshold = failure_threshold  # failures in a row that trip it
        self.reset_after = reset_after              # seconds to stay open
        self.failures, self.state, self.opened_at = 0, "closed", 0.0

    def call(self, fn, *args, **kwargs):
        if self.state == "open":
            if time.monotonic() - self.opened_at < self.reset_after:
                raise CircuitOpen("failing fast")   # no remote call at all
            self.state = "half-open"                # let a trial call through
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self.failures += 1
            if self.state == "half-open" or self.failures >= self.failure_threshold:
                self.state, self.opened_at = "open", time.monotonic()
            raise
        self.failures, self.state = 0, "closed"    # success closes the circuit
        return result

payments = CircuitBreaker()
# payments.call(provider.authorise, order_id, amount, timeout=2.0)

Bulkheads

A ship's hull is divided by bulkheads into watertight compartments, so that a breach floods one compartment rather than sinking the ship. In software a bulkhead partitions resources so that one failing or overloaded part cannot consume everything. The commonest form gives each dependency its own bounded thread pool, connection pool or semaphore: payments may use at most 40 concurrent calls per instance, restaurant lookups 40, courier assignment 40. When payments slow down, their pool fills and further payment calls are rejected at once, but restaurant and courier calls carry on. A timeout limits how long each call can hold resources; a bulkhead limits how many can be held at once, so it protects even when timeouts are set too long.

Bulkheads also work at larger scales. Separate deployments of the same service for different callers (one pool of order-service instances for the consumer app, another for the restaurant-partner API) stop a misbehaving partner integration from starving customers. Cell-based architectures, the Z axis of the scale cube, make each cell a bulkhead for a slice of customers, and shuffle sharding, described by AWS engineers, assigns each customer a random small combination of instances so that one customer's poison traffic affects very few others. The cost of every bulkhead is utilisation: capacity reserved for one compartment cannot be lent to another, so pools must be sized from measured demand.

Fallbacks and graceful degradation

When a call fails or a breaker is open, the caller must still respond. A fallback decides what it returns. The options, from best to worst for the user, are: a cached or stale value (the last known menu, with prices confirmed at checkout); a sensible default (an estimated delivery time of 30 to 45 minutes instead of a live estimate); a reduced feature (popular dishes instead of personalised recommendations); deferring the work (accepting a review and posting it when the review service recovers); and a clear, fast error that tells the user what to do. Designing these per feature, with the product owner, is what graceful degradation means in practice: the core journey (browse, order, pay, track) keeps working while non-essential features step aside. The same thinking guides load shedding (Section 9.3): know in advance which features to switch off first.

Security decisions are the exception: if the authorisation service is unreachable, the safe fallback is to deny (fail closed), not to allow. Practitioners also disagree about fallbacks in general. Nygard and Netflix's engineers advocate them widely. Engineers at Amazon have argued, in the Builders' Library article Avoiding fallback in distributed systems, that fallback paths are rarely exercised, hard to test, and can themselves trigger outages by switching a large fleet to an untested mode at the worst moment; they prefer making the primary path more reliable and failing fast. Both positions agree on the practical rule: a fallback must be simpler than the thing it replaces, must not depend on the component that failed, and must be exercised regularly, for example in game days (Section 9.8), or it should not exist.

Idempotency: what makes retries and hedges safe

Idempotency is the property that doing something twice has the same effect as doing it once. Every pattern in this section that repeats work depends on it: retries after a timeout, hedged requests, message redelivery after a consumer crash, and replays from a dead-letter queue. Section 6.5 showed how an idempotency key turns POST /orders into a safe retry, and Section 6.4 how idempotent consumers deduplicate messages. The architectural rule is simple: before adding a retry, check that the operation is idempotent, and if it is not, add a key or a duplicate check first. A retry policy on a non-idempotent payment call is not a resilience pattern; it is a double-charging generator.

Hedged requests

Retries deal with failures; hedged requests deal with slowness. Section 2.4 introduced them from Dean and Barroso's The Tail at Scale (2013): send a request to one replica, and if it has not answered within, say, the 95th-percentile latency, send a copy to a second replica and use whichever reply comes first, cancelling the other. Because only the slowest few per cent of requests are duplicated, the extra load is small, while the tail shrinks dramatically, as the benchmark quoted in Section 2.4 showed.

Hedging has preconditions. The operation must be idempotent, normally a read. There must be independent replicas, so that the second copy is unlikely to be slow for the same reason. The extra requests must be budgeted and switched off under overload, because when every replica is slow, hedging doubles the load on a system that is already struggling, exactly like an unbudgeted retry. Used carefully on read-heavy, fan-out paths such as search and menu lookups, it is one of the most effective tail-latency tools available.