Be Wise Apps
Leer en español · Lire en français · Auf Deutsch lesen
Section 2.5 of 2.8 · Intermediate · 22 min read
Availability and reliability: nines, MTBF, MTTR and fault tolerance
Video lesson
5 min · narrated, with captionsListen only (audio)
Read the transcript
This lesson is about keeping a service running: how to describe the goal precisely, how to put a number on it, and where an architect's effort pays off most.
Think of a corner shop. Availability is how often the door is open when you arrive. Reliability is whether, once you're inside, the till works all the way through your purchase. Shops stay open by having a spare till, a second member of staff and a quick way to notice and fix problems. Software architects make the same kinds of decisions.
Availability is the proportion of time, or of requests, for which the system is ready to serve. Reliability is the probability that it runs correctly, without failure, for a stated period. They're different. Imagine a service that crashes once an hour and restarts in one second. Its availability is ninety nine point nine seven per cent, which looks excellent. But the chance of a two-hour batch job running without a failure is only about thirteen and a half per cent. A payment terminal cares about availability; a surgical robot cares about reliability.
Three more words matter. A fault is the cause, such as a bug in date code. It can lie dormant for years. When it's activated, it produces an error: wrong internal state. If the error reaches users, it becomes a failure. Every link in this chain is a place to act. Prevent tactics stop faults arising. Detect tactics, such as heartbeats and health checks, notice errors early. And recover tactics, such as failover and retry, stop errors becoming failures.
Availability targets are written in nines. Always convert them into allowed downtime: one minus the availability, times the period. Three nines, ninety nine point nine per cent, allows about eight point seven six hours a year, or forty three point two minutes a month. Four nines allows about fifty two point six minutes a year: just four point three two minutes a month. That's less time than it takes to page an engineer, wake them and work out what's wrong. So above three or four nines, recovery must be automatic.
Availability links two averages: mean time between failures, or MTBF, and mean time to repair, or MTTR. A food-delivery order service fails every thirty days, that's seven hundred and twenty hours, and takes four hours to fix by hand. That's ninety nine point four five per cent: about forty eight point four hours down a year. Option one: double the time between failures. Availability rises to ninety nine point seven two per cent, about twenty four point three hours a year. Option two: automate detection and failover, cutting repair time to fifteen minutes. Availability reaches ninety nine point nine seven per cent, about three hours a year. Only option two beats the target of three nines. Making failures rarer fights physics. Recovering faster is mostly within the architect's control.
Real systems are chains. If a request needs the web tier, the app server and the database, all three must work. They're in series, so their availabilities multiply. Here the chain gives ninety nine point eight four per cent, about fourteen hours down a year: lower than any single part. Now add a second app server in parallel. The pair fails only if both fail at once, so you multiply their unavailabilities. The whole chain rises to ninety nine point nine four per cent, about five point three hours a year. But only if the copies fail independently, and failover really works. A standby that's never been tested is a hope, not a component.
Redundancy in one data centre won't save you if you lose the whole data centre. Disaster recovery uses two numbers from the business. The recovery time objective, or RTO, is the longest acceptable time to restore the service. The recovery point objective, or RPO, is the most data you may lose, measured in time. An RPO of fifteen minutes means losing at most the last fifteen minutes of changes. Test your restores and time them. A backup you've never restored is not a backup. Finally, a service level objective, or SLO, turns availability into a budget. A monthly target of ninety nine point nine per cent allows about forty three minutes of outage, which teams can spend on releases and experiments.
The full section has the complete nines table, the availability tactics and the two shapes of redundancy. Next, in Section two point six, we look at designing for change. See you there.
What you need to know
- Availability is the fraction of time or requests for which the service is ready; reliability is the probability of running without failure for a stated period; they are different and a system can score well on one and badly on the other.
- A fault is a cause, an error is wrong internal state and a failure is a deviation users can observe; tactics prevent faults, detect errors or stop errors becoming failures.
- Allowed downtime = (1 - availability) x period: 99.9% is about 8.76 hours a year or 43.2 minutes a month; 99.99% is about 52.6 minutes a year.
- A = MTBF / (MTBF + MTTR), so cutting repair time through automation and redundancy is often cheaper and more effective than making failures rarer.
- Availabilities multiply in series; in parallel, unavailabilities multiply, but only if the copies fail independently and failover works.
- RTO (how long to restore) and RPO (how much data you may lose) come from the business and drive disaster recovery design and cost.
- SLIs measure, SLOs set internal targets and SLAs are contracts; an SLO turns availability into an error budget that teams can spend on change.
Four words that are often confused
Users experience a system as either working or not. Architects need finer words, because each word points at a different design response. Availability is the proportion of time (or of requests) for which the system is ready to deliver its service when asked. Reliability is the probability that the system performs correctly, without failure, for a stated period under stated conditions. Fault tolerance is the ability to keep delivering the service even though some part has failed. Resilience is the broader capacity to absorb disruption, keep the essential service going (perhaps degraded) and recover to normal afterwards.
ISO/IEC 25010:2023 places availability, fault tolerance and recoverability, together with faultlessness (which replaced the 2011 term maturity), as sub-characteristics of reliability. Bass, Clements and Kazman, in Software Architecture in Practice, treat availability as its own quality attribute because it drives so many architectural decisions. Neither view is wrong; use whichever helps the conversation, but be explicit about which word you mean.
The difference between availability and reliability matters in practice. Imagine a service that crashes once an hour and restarts in one second. Its availability is 3,599 / 3,600 = 99.97%, which looks excellent. Its reliability over a two-hour batch job is poor: if failures arrive at random at a rate of one per hour, the chance of running two hours without one is e-2, about 13.5%. A payment terminal cares about availability; a long-running scientific calculation, a surgical robot or an aircraft cares about reliability over the mission.
Fault, error and failure
The dependability community (Avižienis, Laprie, Randell and Landwehr, 2004) separates three ideas that everyday speech blurs. A fault is the cause: a bug, a failed disk, a misconfiguration, an expired certificate. A fault can lie dormant for years. When it is activated it produces an error: a part of the system's internal state that is wrong, such as a corrupted record or an incorrect value in memory. If the error reaches the service boundary, the result is a failure: the service delivered to users deviates from what it should be.
Bass, Clements and Kazman use the same idea more simply: a failure is a deviation from the specification that the system's users can observe, and the cause of a failure is a fault. The point for architects is that every link in the chain is a place to intervene. You can stop faults arising (code review, removing a component from service before it wears out), detect errors before they spread (validation, health checks), and contain or mask errors so that they never become failures (redundancy, retries, transactions).
The chain and where tactics act
Counting the nines
Availability targets are usually written as a percentage with a number of nines: 99.9% is three nines. It helps to convert every target into allowed downtime, because a percentage hides how little time you really have. The calculation is simple: downtime = (1 - availability) x period. For a 365-day year that is 525,600 minutes, so 99.9% allows 0.001 x 525,600 = 525.6 minutes, about 8.76 hours. For a 30-day month (43,200 minutes) it allows 43.2 minutes.
Each extra nine divides the allowance by ten, while the engineering cost usually rises much faster. At 99.99% you have about 4.3 minutes a month: less than the time it takes for a person to be paged, wake up, log in and understand what is wrong. Above three or four nines, recovery must therefore be automatic; humans are too slow to be in the loop.
Allowed downtime for common availability targets (365-day year, 30-day month)
| Availability | Per year | Per month | Per week | Per day |
|---|---|---|---|---|
| 99% (two nines) | 3.65 days (87.6 h) | 7.2 hours | 1.68 hours (100.8 min) | 14.4 minutes |
| 99.9% (three nines) | 8.76 hours | 43.2 minutes | 10.1 minutes | 1.44 minutes |
| 99.95% | 4.38 hours | 21.6 minutes | 5.04 minutes | 43.2 seconds |
| 99.99% (four nines) | 52.6 minutes | 4.32 minutes | 1.01 minutes | 8.64 seconds |
| 99.999% (five nines) | 5.26 minutes | 25.9 seconds | 6.05 seconds | 0.86 seconds |
MTBF, MTTF and MTTR
Three averages connect failures and repairs. Mean time to failure (MTTF) is the average time a component runs before it fails. Mean time to repair (MTTR) is the average time from failure to restored service (some organisations read the R as recover or restore; the idea is the same, but say which clock you use: from failure, from detection, or from the first alert). Mean time between failures (MTBF) is the average time from one failure to the next.
The classic steady-state formula is A = MTBF / (MTBF + MTTR). Strictly, for a repairable system MTBF = MTTF + MTTR, and the exact expression is MTTF / (MTTF + MTTR): uptime divided by uptime plus downtime. Textbooks often write MTBF where they mean mean uptime; when repairs are short compared with the time between failures, as they usually are, the difference is negligible. The formula assumes a long-run average and says nothing about how outages are distributed, which is why it pairs well with a reliability measure and with percentiles of repair time.
Worked example
Worked example: is it cheaper to fail less often or to recover faster?
An order service for a food-delivery app fails on average once every 30 days (MTBF = 720 hours). Each failure takes 4 hours to repair, because an engineer must be paged, diagnose the problem and restore a database by hand. The business wants better than 99.9%.
| Current availability | 720 / (720 + 4) = 720 / 724 = 0.99448, so 99.45% |
|---|---|
| Current downtime per year | (1 - 0.99448) x 8,760 h = about 48.4 hours |
| Option 1: double MTBF | Better hardware and testing; failures every 60 days. A = 1,440 / 1,444 = 0.99723 (99.72%), about 24.3 hours a year |
| Option 2: cut MTTR to 15 minutes | Automated detection, a hot standby and scripted failover. A = 720 / 720.25 = 0.99965 (99.97%), about 3.0 hours a year |
| Target check | Only option 2 beats 99.9% (8.76 hours a year) |
Halving how often failures happen halved the downtime. Cutting repair time by a factor of 16 cut the downtime by about the same factor, and passed the target.
Raising MTBF fights physics and complexity: you cannot make disks, networks or people perfect. Lowering MTTR is mostly within the architect's control: detection, automation, redundancy, small reversible deployments. This is why modern operations practice concentrates on recovery.
MTTR is an average and hides long tails. One eight-hour outage among many five-minute ones dominates the year, so track the distribution of incident durations, not only the mean.
Components in series and in parallel
Real systems are chains of parts. If a request needs the web tier, the application server and the database, all three must work: the components are in series, and their availabilities multiply. Because every factor is below one, the product is always lower than the weakest link. Ten components at 99.9% each give 0.99910 = 0.990, only 99.0%.
If instead any one of several identical copies is enough, the copies are in parallel (redundancy). The pair fails only when every copy fails at once, so you multiply the unavailabilities: Aparallel = 1 - (1 - A)n for n identical copies. Two servers at 99% each give 1 - 0.01 x 0.01 = 99.99%.
Series and parallel with numbers
Worked example
Worked example: where should the next pound be spent?
Continuing the diagram. The chain in part A is 99.84%. Which change brings it closest to 99.99%?
| Add a second app server | 0.9999 x 0.999999 x 0.9995 = 0.99940 (99.94%): about 5.3 hours a year |
|---|---|
| Then add a database replica with failover | Database pair: 1 - 0.00052 = 0.99999975. Chain: 0.9999 x 0.999999 x 0.99999975 = 0.99990 (99.99%): about 53 minutes a year |
| What now limits the system | The single web tier (99.99%) contributes almost all of the remaining downtime |
The arithmetic assumes the copies fail independently. Two servers in the same rack, on the same power feed, running the same buggy release, or both depending on one DNS provider, share common-mode faults and can fail together. Real gains are always smaller than the formula suggests.
It also assumes failover is instant and always works. A standby that has never been tested in a real failover is a hope, not a component.
A useful rule from Google's SRE authors (Treynor, Dahlin, Rau and Beyer, 2017): each critical dependency should offer roughly one more nine than the service that relies on it, or the service needs a way to cope without it.
Availability tactics: detect, recover, prevent
Bass, Clements and Kazman group availability tactics into three families. They are design options, not a checklist to apply in full; each has a cost.
The three families of availability tactics
- Detect faultsHeartbeat: a component sends a periodic "I am alive" message, and silence triggers action. Ping/echo: a monitor asks and expects a reply. Health checks: an endpoint such as
/healththat a load balancer or orchestrator polls (a liveness check asks "is the process running?"; a readiness check asks "can it serve traffic now?"). Monitoring of error rates and latency, sanity checks on values, and voting between replicas that compute the same result. - Recover: preparation and repairRedundant spares in three temperatures: active-active (all copies serve live traffic), hot or warm standby (a copy kept in step and ready), and cold spare (started only when needed). Failover switches traffic to a healthy copy. Retry handles transient faults (with limits and back-off, or retries become a self-inflicted overload). Rollback returns to a known good state or release. Graceful degradation keeps the core service going without a failed feature: the shop still takes orders even if recommendations are down.
- Recover: reintroductionBringing a repaired component back safely: run it in shadow mode, resynchronise its state, restart in escalating steps (a thread, then a process, then a machine).
- Prevent faultsRemoval from service (take a node out of rotation for maintenance or a scheduled restart before a known memory leak bites), transactions (all-or-nothing updates so a crash cannot leave half-written state), predictive models (replace a disk when its error counters climb), and exception prevention (stronger types, input validation).
Two shapes of redundancy
Choosing between active-passive and active-active
Active-passive is simpler to reason about, especially for stateful components such as a relational database with one writable primary. Its costs are the idle standby, a failover gap (detection plus promotion, often tens of seconds), and the risk that the standby has quietly drifted out of step. A classic trap is split brain: the network between the two breaks, each believes the other is dead, and both accept writes. Guarding against it needs a quorum or an external arbiter.
Active-active uses all capacity and has no promotion step, but every node must be sized to carry the full load if its partner fails. Two nodes each running at 70% utilisation will be asked for 140% when one dies, and the survivor collapses: a redundant design that fails anyway. Active-active also forces you to solve data consistency across copies, which is a large topic in itself (see consistency, CAP and PACELC).
Disaster recovery: RTO and RPO
Redundancy within one data centre does not protect you from losing the whole data centre, a region, a cloud account or your data to ransomware. Disaster recovery planning uses two numbers that should come from the business, not from engineers. The recovery time objective (RTO) is the longest acceptable time to restore the service after a disaster. The recovery point objective (RPO) is the largest acceptable loss of data, measured in time: an RPO of 15 minutes means you may lose at most the last 15 minutes of changes.
The two numbers drive very different designs and costs. Nightly backups copied to another region give an RPO of up to 24 hours and an RTO of many hours (restoring terabytes takes time). Continuous asynchronous replication to a warm standby in another region might give an RPO of seconds and an RTO of minutes, at perhaps double the infrastructure cost. Synchronous replication can bring the RPO close to zero but adds the round-trip time between sites to every write. Cloud providers describe a ladder of strategies: backup and restore, pilot light, warm standby and multi-site active-active, each faster and more expensive than the one before. The details belong to multi-region and disaster recovery.
Whatever the strategy, a backup you have never restored is not a backup. Test restores on a schedule and time them; the measured restore time is your real RTO.
From targets to SLIs, SLOs and SLAs
Modern operations practice expresses availability with three related terms, covered properly in SLIs, SLOs and error budgets. A service level indicator (SLI) is the measurement, for example the proportion of checkout requests that succeed in under 800 ms. A service level objective (SLO) is the internal target for that measurement over a window, for example 99.9% over 30 days. A service level agreement (SLA) is a contract with customers that names consequences, such as service credits, if a level is missed; it is normally set looser than the SLO so the team has warning before the contract is breached.
The SLO turns availability into a budget. A 99.9% monthly SLO allows 0.1% of requests to fail; at 2 million requests a month that is 2,000 failed requests, or about 43 minutes of full outage. Teams spend this error budget on risk: releases, experiments, migrations. When it is gone they slow down and invest in reliability. This links the abstract nines in this section to day-to-day decisions.
Common mistakes
Promising nines without counting dependencies: a 99.99% service built on a single 99.9% database cannot meet its target. Adding redundancy without removing shared causes (the same release, the same configuration push, the same certificate expiring on every node at midnight). Measuring availability from inside the system, where everything looks green, while users cannot reach it because of DNS or a content delivery network. Planning failover but never rehearsing it; chaos engineering exists precisely to exercise these paths deliberately (chaos engineering). Finally, chasing a number for its own sake: a staff rota system used in office hours does not need five nines, and the money is better spent elsewhere.
To make availability testable, write it as a scenario (Section 2.3): When the primary database server fails during peak hours, the system detects it within 10 seconds and resumes accepting orders within 60 seconds, losing no confirmed order.
Every part of that sentence becomes a design choice and a test.
Check your understanding
How many minutes of downtime does a 99.95% target allow in a 30-day month?
0.0005 x 43,200 = 21.6 minutes.
Two independent load balancers each have 99.5% availability and either can carry all traffic. What is the availability of the pair?
1 - (0.005 x 0.005) = 1 - 0.000025 = 99.9975%.
Why might adding a second identical server give much less benefit than the parallel formula predicts?
Because the copies may share common causes of failure, such as the same release, configuration, power, network or dependency, so they do not fail independently; failover itself may also be slow or broken.
A team's backups run every 6 hours and a full restore takes 3 hours. What are their effective worst-case RPO and RTO?
RPO up to 6 hours of lost data, and an RTO of at least 3 hours plus the time to detect the disaster and decide to restore.
Sources and further reading
- Google SRE book: Embracing Risk (time-based and request-based availability)
- Google SRE book: Service Level Objectives
- SEI blog: Tactics and Patterns for Software Robustness (availability tactics)
- AWS whitepaper: Disaster recovery options in the cloud
- Avižienis, Laprie, Randell and Landwehr: Basic Concepts and Taxonomy of Dependable and Secure Computing (IEEE TDSC, 2004)
Have a question about this section? ArchitectBot, the course’s AI study assistant, answers from the course first.
Ask: “Our checkout depends on a payment provider with a 99.9% SLA. How can we offer our own customers 99.95% availability?”