Chaos engineering, incidents and blameless learning

Practising failure on purpose

Everything in this chapter so far is a promise about how the system will behave when something goes wrong: timeouts will fire, circuit breakers will open, load will be shed, replicas will take over, alerts will reach someone who knows what to do. Chapter 2 warned that failover which has never been exercised often fails when it is finally needed (Section 2.5). This section is about turning those promises into evidence, and about how organisations respond to, and learn from, the failures that happen anyway.

Two practices belong together. Before failures, chaos engineering and game days deliberately cause small, controlled failures to find weaknesses and to train people. After failures, incident management limits the damage and blameless post-incident reviews turn each incident into lasting improvement. Both rest on the same belief: complex systems are always partly broken, and resilience comes from finding out how, early and safely.

Chaos engineering

Chaos engineering grew out of Netflix's migration from its own data centres to Amazon Web Services, which began after a serious database corruption in 2008. Individual cloud instances could disappear at any time, so Netflix built Chaos Monkey, a tool that randomly terminated production instances during business hours, when engineers were at their desks to notice and fix any resulting problem. Netflix described it, with a family of related tools called the Simian Army, in a July 2011 blog post. Later tools went further: Chaos Kong simulated the loss of an entire AWS region by evacuating traffic from it.

Netflix engineers and others later wrote down the discipline as the Principles of Chaos Engineering (principlesofchaos.org, last revised in 2019), which define it as experimenting on a system to build confidence in its capability to withstand turbulent conditions in production. The word that matters is experimenting: chaos engineering is not breaking things at random to see what happens, but the scientific method applied to resilience. The principles list five practices:

Build a hypothesis around steady-state behaviour. Choose a measurable output that indicates normal operation, such as orders placed per minute or the checkout SLI, rather than internal metrics like CPU. The steady-state hypothesis predicts that this output will stay within bounds while the failure is injected. Vary real-world events. Inject the failures that actually happen: instances dying, a dependency slowing down, packet loss, a full disk, an expired certificate, clock skew, a spike in traffic. Run experiments in production. Only production has the real traffic, data and configuration. Automate experiments to run continuously. A one-off experiment proves resilience once; systems change daily. Minimise the blast radius. Start with the smallest experiment that can teach something (one instance, a small percentage of traffic) and keep an abort switch ready.

Running experiments in production is the most debated principle. Its defenders argue that staging environments differ in traffic, data, scale and configuration, so only production experiments reveal real weaknesses, and that a small, controlled experiment at 15:00 is far kinder to users than the same failure arriving uncontrolled at 03:00. Critics point out that many organisations lack the observability, automation and organisational trust to do this safely. The practical consensus is a progression: begin in a staging environment, then run in production with a tiny blast radius and automatic abort conditions tied to SLOs, and widen only as confidence grows. Never start with a system you already know is fragile; fix the known weaknesses first, because an experiment whose outcome you can predict teaches nothing.

Tooling is now widely available: cloud providers offer managed fault injection (AWS Fault Injection Service, Azure Chaos Studio), Chaos Mesh and LitmusChaos are open-source CNCF projects for Kubernetes, and commercial platforms such as Gremlin provide libraries of attacks with safety controls. Service meshes and proxies can inject latency and errors into a percentage of calls without touching application code. The tools are the easy part; the hard part is choosing meaningful hypotheses and acting on the results.

Game days

A game day is a planned exercise in which a team deliberately causes or simulates a significant failure and practises responding to it. Jesse Robbins popularised the practice at Amazon, where his job title was reported as Master of Disaster. Google runs large Disaster Recovery Testing (DiRT) exercises, and its SRE book describes the Wheel of Misfortune, a role-play in which on-call engineers work through a past incident while a game master reveals what the monitoring would have shown.

Game days test more than software. They test whether alerts reach the right people, whether people have the access they need at 03:00, whether runbooks are current, whether the failover procedure actually works, whether backups restore, and whether the communications plan reaches customers. A typical food-delivery game day might fail over the order database to its standby during a quiet afternoon, with the whole on-call rotation watching, a facilitator injecting surprises (the person who knows the runbook is on leave) and an observer noting every delay and confusion for the review afterwards. Before a big ticket on-sale, the same team might rehearse losing the payment provider and switching to the backup provider mid-sale.

Incident management

An incident is an unplanned event that disrupts, or threatens to disrupt, a service and needs a coordinated response. Incident management is the practice of running that response so that it is fast, calm and well communicated. Most modern practice borrows from the Incident Command System, developed for wildfire response in California in the 1970s, and adapted for software in the Google SRE book's chapter on managing incidents.

The central idea is separation of roles. The incident commander owns the incident: sets priorities, assigns roles, makes decisions and keeps the overview, and deliberately does not debug, because a commander buried in a log file is not commanding. The operations lead and responders diagnose and mitigate. The communications lead is the single voice to everyone outside the response: support teams, executives and customers through the status page, with updates at a promised cadence even when there is nothing new. A scribe records the timeline, decisions and hypotheses. In a small company one person may hold several roles, but the roles should still be named. Roles are handed over explicitly when shifts change.

Several habits make the difference. Declare incidents early and cheaply: it is better to stand down a false alarm than to discover an hour in that three teams have been debugging the same problem separately. Classify each incident with a severity level defined by user impact, not by cause, so that everyone knows who to call and how widely to communicate. Mitigate before diagnosing: roll back the last deployment, fail over, switch off the feature flag or shed load first, and find out why later. Use one incident channel. And remember external obligations: a personal-data breach may have to be reported to the regulator within 72 hours under the GDPR, which is why security incidents follow the related process in Section 8.8.

Blameless post-incident reviews

After the incident comes the most valuable part: learning. A post-incident review (often called a postmortem) reconstructs what happened, why it made sense to the people involved at the time and what will change. The Google SRE book's chapter on postmortem culture sets out the expectations: a written record for every significant incident, covering impact, timeline, contributing causes and follow-up actions, reviewed and shared, and written without blame.

John Allspaw's 2012 post Blameless PostMortems and a Just Culture, written when he led technical operations at Etsy, explains why blamelessness is practical rather than merely kind. Engineers who expect punishment stop giving full accounts of what they saw and did, so the organisation loses exactly the information it needs to prevent the next incident. Allspaw draws on safety science, including Sidney Dekker's work on just culture, and on the idea of second stories: the first story says an engineer ran the wrong command; the second asks why the command was easy to run, why the tool did not warn, why the procedure seemed reasonable under time pressure, and finds conditions that will trap the next person too. Blameless does not mean unaccountable: people are still expected to give a full account, and the organisation is accountable for fixing the conditions.

Good reviews avoid hindsight bias (the outcome was not obvious at the time), counterfactuals (should have, failed to) and the search for a single root cause. Richard Cook's short paper How Complex Systems Fail (1998) argues that serious failures in complex systems need several flaws to line up, so there is no single root cause, only contributing factors. A five-whys chain that ends at human error has stopped too early. Note also what went right: the expertise and improvisation that limited the damage are part of how the system really stays up, and they are invisible unless someone records them.

Learning from incidents, and the limits of MTTR

Organisations naturally want a number to show whether incident response is improving, and mean time to recovery (MTTR) is the usual candidate. Chapter 2 used MTTR in A = MTBF / (MTBF + MTTR), where it is meaningful for a single component with repeatable failures. As an organisational metric across many different incidents it is much weaker. Incident durations are highly skewed: most are short and a few are very long, so the mean is dominated by a handful of outliers. Courtney Nash's analyses of thousands of public incident reports in the Verica Open Incident Database (VOID reports from 2021 onwards) found exactly this skew, and a 2021 Google report by Štěpán Davidovič, Incident Metrics in SRE, used simulations to show that, with realistic numbers of incidents, changes in MTTR are mostly noise and cannot reliably show whether an improvement worked.

Better signals are the impact on SLOs and error budgets (which weight incidents by what users felt), recurring themes across reviews (the same kind of configuration change causing trouble three times), near misses, how quickly incidents are detected, how often reviews lead to completed system changes, and qualitative evidence such as whether engineers find reviews useful. Many organisations publish reviews internally in a searchable library and hold regular incident review meetings, so that one team's lesson reaches others.

Action items that actually change the system

A review is worth only the changes it produces. Actions differ greatly in strength. Strongest are those that remove the hazard (delete the dangerous command, make the unsafe state impossible), then guardrails that stop the mistake propagating (validation of configuration, staged rollouts and canaries, rate limits on changes, automatic rollback on SLO burn), then faster detection and mitigation (a new SLO alert, a tested runbook, an automated failover). Weakest are training, documentation and reminders. Be more careful is not an action item; it changes nothing about the conditions that will meet the next person.

Each action needs an owner, a priority and a due date, and must live in the team's normal backlog rather than in a forgotten document. The error budget policy from Section 9.5 provides the leverage: an incident that consumed a large share of the budget earns its fixes priority over features. Keep the list short; five completed actions beat twenty open ones. Where an action changes the architecture, such as adding a bulkhead or removing a synchronous dependency, record it in an architecture decision record that cites the incident, and link the relevant runbooks from the alerts that will fire next time.