SLIs, SLOs, SLAs and error budgets

Three terms, one idea

Chapter 2 introduced availability targets and the nines (Section 2.5) and showed that 99.9% over 30 days leaves 43.2 minutes. This section turns that arithmetic into an operating discipline. The vocabulary comes from Google's Site Reliability Engineering book (Beyer, Jones, Petoff and Murphy, O'Reilly, 2016) and its companion, The Site Reliability Workbook (2018), and is now used far beyond Google.

A service level indicator (SLI) is a carefully defined, quantitative measure of some aspect of the service users receive. The most useful form is a ratio: good events divided by valid events. The proportion of checkout requests that return a successful response within 800 ms, measured at the load balancer is an SLI. It ranges from 0% (nothing works) to 100% (nothing is broken), and the ratio form means every SLI can be read, compared and alerted on the same way.

A service level objective (SLO) is a target for an SLI over a stated window: 99.9% of checkout requests are good, measured over a rolling 28 days. It is an internal promise that the team, the product owner and the people who depend on the service agree to. A service level agreement (SLA) is a contract with customers that names consequences, usually service credits, if a level is missed. Many services have SLOs but no SLA; internal platforms almost never have SLAs at all.

The idea that ties the three together is the error budget. If the target is 99.9%, then 0.1% of events may be bad without anyone having failed. That 0.1% is not waste; it is the room the team has to take risks: releasing features, running migrations, experimenting with configuration. The budget turns an argument about feelings (is the service reliable enough?) into an argument about numbers (have we spent our budget?), and gives product and engineering a shared rule for when to move fast and when to slow down.

Choosing good SLIs

An SLI is only as useful as its closeness to what users experience. The Workbook offers a short menu that covers most services. Request-driven services (APIs, web pages) need availability (the proportion of valid requests served successfully), latency (the proportion served faster than a threshold) and sometimes quality (the proportion served without degradation, such as a menu shown without personalised recommendations because that dependency was down). Data pipelines need freshness (the proportion of data updated more recently than a threshold), correctness (the proportion of output that is right) and coverage (the proportion of input processed). Storage systems add durability (the proportion of written data that can still be read).

Latency SLIs are written as proportions, not averages: 99% of menu requests faster than 300 ms. This connects directly to the percentiles of Section 2.4: saying 99% of requests must be under 300 ms is the same as saying p99 must be under 300 ms, but the proportion form adds up cleanly across time and servers (count the good ones and the valid ones), while percentiles do not. It is common to set two latency thresholds, for example 90% under 200 ms and 99% under 800 ms, to capture both typical and tail experience.

Where you measure matters as much as what you measure. The Workbook separates the SLI specification (the user-relevant outcome, such as orders placed successfully) from the SLI implementation (how you count it: server logs, load balancer metrics, synthetic probes or instrumentation in the mobile app). Measuring inside the application server misses requests that never reached it because DNS, the content delivery network or the load balancer failed. Measuring in the client sees everything but includes users' bad mobile connections and arrives late. Most teams start with load balancer metrics, which are cheap and close to the edge, and add synthetic probes for low-traffic paths.

Define what counts as valid and what counts as good precisely, because the edge cases decide whether the number means anything. Is a 404 for a mistyped URL bad? Usually not: it is a correct answer. Is a 429 rate-limit response bad? It depends on whether the client was within its agreed quota. Are health checks and load tests valid events? No; exclude them, or they dilute the ratio. Write these rules down next to the SLO.

Setting the target

A target is a business decision, not a technical one. The cost of each extra nine rises steeply, while users stop noticing improvements beyond some point because their own phone network, Wi-Fi and devices fail more often than that. Google's SRE book makes the point bluntly: aiming for 100% is wrong for almost everything, because it leaves no budget for change, costs a fortune and cannot be achieved anyway. A 100% target also turns every single failure into a policy breach, which teaches people to ignore the SLO.

Good starting points are past performance (what did the service achieve over the last quarter, and were users happy?), the targets of the services you depend on (you cannot promise 99.99% on top of a 99.9% database, as Section 6.1 showed for chains), and the business cost of unreliability. For the food-delivery platform, failed orders at lunchtime cost revenue and customer trust, so 99.9% on order placement is defensible; courier position updates are less critical and 99% is enough. Begin a little below current performance, so the team is not instantly in breach, and tighten later with evidence.

Choose the window deliberately. A rolling window such as the last 28 days matches how users remember outages (they do not forget on the first of the month), and 28 days contains exactly four of each weekday, which keeps weekly patterns fair. Calendar months suit reporting and contracts. Event-based services such as a ticket on-sale may need an SLO per event, because an hour of failure during the on-sale matters far more than a quiet week.

Error budgets in numbers

The error budget can be counted in time or in events. Time-based budgets suit services that are either up or down; request-based budgets are fairer for services with uneven traffic, because an outage at 03:00 affects fewer users than one at noon.

Error budget policies

A budget means nothing unless something happens when it runs out. An error budget policy is a short document, agreed in advance by engineering, product and leadership, that says what the team will do. The example policy in the Workbook measures the budget over a trailing four weeks; if the service exceeds its budget, all changes and releases stop except urgent fixes (P0 issues) and security fixes until the service is back within its SLO. It also requires a postmortem with at least one top-priority action item whenever a single incident consumes more than 20% of the four-week budget, and escalates disagreements to the chief technology officer.

The value of a policy lies in being agreed before it bites. In the middle of a release crunch, nobody wants to freeze; with a signed policy, the freeze is a pre-agreed rule rather than a fight between an SRE and a product manager. Good policies also say what does not count against the team: a company-wide network outage, a dependency that is itself frozen, or traffic outside the SLO's scope such as a load test.

Practitioners disagree about how mechanical the policy should be. Some teams apply it strictly, because exceptions erode trust in the whole system. Others treat the budget as information for a conversation, arguing that a hard freeze can delay the very fixes that would restore reliability. A middle path is common: freeze risky feature releases automatically, keep shipping reliability fixes, and allow a named executive to grant documented exceptions. The opposite case also matters: a team that never spends its budget is probably too cautious and could ship faster, run more experiments or lower its costs.

Burn rate and alerting on SLOs

Alerting on raw error rates (page if more than 1% of requests fail for 5 minutes) is noisy: short blips page people at night for problems that would never threaten the SLO, while a steady 0.5% failure rate can quietly use up the month without ever crossing the threshold. The Workbook's answer is to alert on the burn rate: how fast the budget is being spent compared with the rate that would use it up exactly at the end of the window.

Burn rate = observed error ratio / allowed error ratio. With a 99.9% SLO the allowed ratio is 0.1%, so an error ratio of 0.1% is a burn rate of 1 (the budget lasts exactly the window), 1% is a burn rate of 10, and 100% (a full outage) is a burn rate of 1,000. The fraction of the window's budget spent during an alerting window is burn rate x alerting window / SLO window. For a 30-day (720-hour) window, a burn rate of 14.4 sustained for one hour spends 14.4 × 1 / 720 = 0.02, that is 2% of the month's budget; at that rate the whole budget would be gone in 720 / 14.4 = 50 hours.

The Workbook recommends a multiwindow, multi-burn-rate alert as a sensible default for a 99.9% SLO: page if 2% of the budget is spent in one hour (burn rate 14.4) or 5% in six hours (burn rate 6), and open a ticket if 10% is spent in three days (burn rate 1). Each condition must hold over a long window and a short one, with the short window one twelfth of the long (5 minutes, 30 minutes and 6 hours respectively). The long window ensures the problem is significant; the short window ensures it is still happening, so the alert stops firing within minutes of a fix instead of staying red for an hour.

SLO = 0.999                 # 99.9% of requests good
PERIOD_H = 30 * 24          # 720-hour SLO window

def burn_rate(bad, total, slo=SLO):
    """Observed error ratio divided by the ratio the SLO allows."""
    if total == 0:
        return 0.0
    return (bad / total) / (1 - slo)

def budget_spent(rate, window_h, period_h=PERIOD_H):
    """Fraction of the whole window's budget used in window_h hours."""
    return rate * window_h / period_h

def page(w1h, w5m, w6h, w30m):
    """Each argument is (bad, total) for that window."""
    fast = burn_rate(*w1h) >= 14.4 and burn_rate(*w5m) >= 14.4
    slow = burn_rate(*w6h) >= 6 and burn_rate(*w30m) >= 6
    return fast or slow

print(round(burn_rate(bad=288, total=20_000), 1))   # 14.4
print(budget_spent(14.4, window_h=1))               # 0.02, i.e. 2%

The thresholds also fix detection time. If the order API fails completely (error ratio 100%, burn rate 1,000), the one-hour error ratio crosses 1.44% after 0.0144 × 60 = about 0.9 minutes, so the page fires in under a minute. If 2% of requests fail (burn rate 20), the one-hour ratio reaches 1.44% after 0.72 × 60 = about 43 minutes. Severe problems page fast; mild ones page only if they persist, which is exactly the behaviour people want at three in the morning.

Low-traffic services break this arithmetic: with 10 requests an hour, one failure is a 10% error ratio and a burn rate of 100. The Workbook's remedies are synthetic traffic to add signal, grouping related low-traffic services under one SLO, making single failures matter less through retries and fallbacks, or accepting a longer window. For the ticketing system, which is almost idle between on-sales and then extremely busy, per-event SLOs with burn-rate alerts during the event window work better than a monthly figure.

SLAs and service credits

An SLA moves the target from engineering into contract law. It defines the service covered, how availability is measured, exclusions (scheduled maintenance, customer misuse, force majeure), the remedy and how a customer claims it. The usual remedy is a service credit: a percentage of the monthly fee returned, tiered by how badly the level was missed. Credits are typically small compared with the customer's actual loss, which is why SLAs are better read as a signal of the provider's confidence than as insurance; customers whose losses would be large need their own resilience, not a bigger credit.

Consider an illustrative SLA the food-delivery platform might offer restaurant partners for its order-intake API: 99.5% monthly availability, with a 10% credit on that month's platform fee below 99.5% and 25% below 99.0%. The internal SLO is 99.9%. The gap between them is deliberate: the team should exhaust its SLO budget, apply its policy and recover long before the contract is at risk. Never set the SLO equal to the SLA; you would learn about trouble only when you already owe money.

Two architectural consequences follow. First, an SLA can only be honest if you know the SLAs of your own dependencies (cloud provider, payment processor, map provider) and how they combine. Second, SLAs need evidence: the measurement method in the contract must be one you can produce from your own telemetry, which is a requirement on the observability design.

Common mistakes

Too many SLOs. A service with thirty SLOs has none, because nobody can reason about them or act on them. Start with two or three per critical user journey: usually one availability and one or two latency SLOs. Measuring the wrong thing. CPU utilisation, queue depth and replica lag are useful causes to watch, but they are not SLIs, because users do not experience them directly. Measuring in the wrong place. Server-side success rates look perfect while the CDN returns errors. Averaging. An average latency SLO hides the slow tail that users complain about.

Targets without consequences. An SLO with no budget policy and no burn-rate alerts is a dashboard decoration. Targets set by engineers alone. The product owner must agree, because the SLO trades features for reliability. Never revising. Review SLOs every quarter or two: if users complain while the SLO is green, the SLI or the target is wrong; if the budget is never touched, the target may be too loose or the team too cautious. One SLO for very different journeys. Lumping menu browsing and payment into one ratio lets plentiful cheap requests hide failures in the scarce, valuable ones.

SLOs connect to the rest of this chapter. Burn-rate alerts are the symptom-based alerts recommended in observability; budget-burning incidents feed post-incident reviews; and an SLO's response measure is a natural fitness function for the architecture.