Be Wise Apps
Leer en español · Lire en français · Auf Deutsch lesen
Section 11.1 of 11.8 · Advanced · 28 min read
Architecture evaluation and the ATAM
Video lesson
5 min · narrated, with captionsListen only (audio)
Read the transcript
Every architecture is a bet. This lesson shows how to check that bet before the money is committed, using the Architecture Tradeoff Analysis Method, or ATAM.
Before a city builds a new bridge, independent engineers check the design. They ask what happens in a flood, in a storm, and under the heaviest lorries. They point out where one choice that saves money makes another danger worse. They don't redesign the bridge or give it a mark. They tell the owners where the real risks are, while changes are still cheap. Architecture evaluation does the same for software.
An evaluation asks a narrow, practical question: given what this system must achieve, which of its architectural decisions put those goals at risk? It works on decisions, not code, so it can start on a whiteboard design. It's most valuable before a major commitment, such as a vendor contract or a cloud migration. Held too late, its findings can only be regretted.
The ATAM was described in 2000 by Rick Kazman, Mark Klein and Paul Clements of the Software Engineering Institute, or SEI. An evaluation team of three to five people from outside the project runs it. Independence matters: evaluators who designed the system find fewer risks. They work with the decision makers, such as the architect and the sponsor, and with a wider group of stakeholders, from developers to operations, security and support. The ATAM gives no score and no pass mark. It finds where decisions put the business goals at risk, and where they force trade-offs between qualities.
The method runs in four phases, with nine steps across two evaluation meetings. In phase zero, over a few weeks, the leaders agree the scope, cost and team, and gather the architecture documents. Phase one takes about a day with the decision makers. Steps one to six present the method, the business drivers and the architecture, then build the utility tree and analyse. After a pause of two to three weeks, phase two adds the wider stakeholders for about two days. They brainstorm and vote on their own scenarios, which are then analysed. Phase three is about a week of follow-up: the final report, with risk themes turned into backlog items and decision records.
The heart of the method is the analysis. The evaluators take the prioritised scenarios, and the architectural approaches that address them. They ask the architect to walk each scenario through the design, and probe. How many synchronous calls are on this path? What happens when that node fails? Each answer is classified. A risk is a decision, or a missing decision, that may stop an important goal being met. A non-risk is a sound decision, recorded with its assumptions. A sensitivity point is a property that one quality depends on critically. A trade-off point is a sensitivity point for more than one quality, so improving one makes another worse. Risks and trade-off points are grouped into risk themes, each with a shared underlying cause. And each theme names the business drivers it threatens, the same drivers that shaped the scenarios.
Here's a condensed ATAM of the food delivery platform, with four thousand restaurants and nine thousand couriers. The business wants to grow from fifty to a hundred orders a second at the Friday peak within eighteen months. Order latency must stay under three seconds at the ninety ninth percentile, but the payment provider alone takes two and a half seconds. By Little's law, that's two hundred and fifty requests in flight for two hundred workers. Risk R1. When the provider times out, the current design fails the order: risk R2. Authorising later, from a queue, would help, but some accepted orders would then fail payment. That's trade-off point T1. Not everything is a risk. The outbox, Kafka with three replicas and idempotent consumers mean no lost orders: non-risk N1, with its assumption recorded. In phase two, stakeholders voted highly for a scenario missing from the tree: a restaurant tablet offline for ten minutes at peak must still get every order. Risk R5, a blind spot. The risks group into two themes: third parties on synchronous critical paths, and recovery and delivery guarantees that lag behind the availability promised to restaurant chains.
A full ATAM is expensive: in the course's rough illustration, around sixty days of people's time. For a system whose failure would cost millions, that's cheap insurance. For a small team it's out of proportion, so scale the method to the stakes. A reduced ATAM can be completed in a day or less. Findings that don't become decision records, backlog items and fitness functions with owners are forgotten within a quarter.
The full section sets out all nine steps, and how the Cost Benefit Analysis Method puts money on the table. Next, in Section eleven point two, we look at lightweight reviews and risk-storming. See you there.
What you need to know
- Architecture evaluation checks whether a system's significant decisions will let it meet its business goals and quality requirements, and is most valuable before major or irreversible commitments.
- The ATAM (Kazman, Klein and Clements, SEI, 2000) runs nine steps across preparation, two evaluation meetings and follow-up, with an independent evaluation team, the project's decision makers and a wider group of stakeholders.
- Its outputs are risks, non-risks (with their assumptions), sensitivity points, trade-off points and risk themes linked to the business drivers they threaten; it does not award a pass mark.
- The utility tree built by decision makers in step 5 is compared with the scenarios stakeholders brainstorm and vote on in step 7, and gaps between them reveal blind spots.
- CBAM builds on the ATAM by weighting the utility each architectural strategy adds to prioritised scenarios and dividing by its cost to compare value for cost, while constraints can still override the ratio.
- A full ATAM costs tens of person-days, so the method should be scaled to the stakes, and its findings must become decision records, backlog items and fitness functions with owners.
Why evaluate an architecture
Every architecture is a bet. When a team chooses an event broker, a database per service or a single cloud region, it is predicting that those decisions will deliver the performance, availability, security and cost the business needs. Section 1.3 showed why such bets matter: the cost of changing a structural decision grows with everything built on top of it, and some of the most expensive failures in software were failures of arrangement rather than of individual lines of code. Architecture evaluation is the deliberate act of checking the bet before, or while, the money is spent.
An evaluation asks a narrow, practical question: given what this system must achieve, which of its architectural decisions put those goals at risk? It is not a code review, a beauty contest between styles or an audit of documentation quality. It works on decisions and their consequences for quality attributes, so it can be done long before there is code, on a whiteboard design or a set of decision records, and it can be repeated on a system that has been running for years.
Evaluation also has benefits beyond the findings. Stakeholders who rarely meet (product owners, operations, security, finance, developers) have to agree what matters most, which often exposes conflicting assumptions. The architect has to explain the design clearly enough for outsiders to probe it, which improves the documentation. And the organisation gets a record of known risks that it has chosen to accept, which is very different from risks nobody noticed.
When to evaluate
The best time to evaluate is when enough has been decided for the evaluation to have something to examine, but before those decisions become expensive to reverse. In practice there are several natural moments, and mature organisations use different weights of evaluation for each.
Moments for evaluation
| Trigger | What is being checked | Typical weight |
|---|---|---|
| Candidate architecture exists, before large-scale build | Whether the main styles and tactics can meet the prioritised quality scenarios | Full or reduced ATAM for large or long-lived systems; a lightweight review otherwise |
| Before a major commitment | A vendor contract, a cloud migration, a funding gate or a re-platforming decision that would be costly to undo | Formal, often with external evaluators, because the decision is a one-way door |
| Significant change in business drivers | New markets, a merger, a tenfold growth target, new regulation | Re-run the utility tree and analyse the scenarios that changed |
| Accumulated change and drift | Whether the system as built still matches the intended architecture and its reasons | Periodic lightweight review plus automated fitness functions |
| Acquisition or due diligence | Whether an existing system can support the buyer's plans | Time-boxed evaluation of an existing system, often combined with code analysis |
| After a serious incident | Whether the incident exposed an architectural weakness rather than a one-off fault | Focused analysis of the affected scenarios |
Evaluating too early wastes effort on decisions that are not yet real; evaluating too late turns findings into regrets. A useful rule from Fairbanks's risk-driven model (Section 1.8) applies here too: invest in evaluation in proportion to the risk. A two-team internal tool does not need a two-day workshop; a national payments platform does.
The ATAM: origins, purpose and participants
The Architecture Tradeoff Analysis Method (ATAM) is the best-known evaluation method. Rick Kazman, Mark Klein and Paul Clements of the Software Engineering Institute (SEI) at Carnegie Mellon University described it in the technical report ATAM: Method for Architecture Evaluation (CMU/SEI-2000-TR-004, August 2000). It grew out of the earlier Software Architecture Analysis Method (SAAM), which Kazman and colleagues published in 1994 and which focused mainly on modifiability. The ATAM is also described in Software Architecture in Practice by Bass, Clements and Kazman, now in its fourth edition (2021).
Its purpose is often misunderstood. The ATAM does not give an architecture a score or a pass mark, and it does not prove that a system will meet its requirements. It identifies where architectural decisions put the business goals at risk and where they force trade-offs between qualities, so that the people responsible can decide what to do. The method's name stresses trade-offs because most interesting risks sit where one quality is bought at the expense of another.
Three groups take part. The evaluation team, usually three to five people from outside the project, runs the method: a team leader, an evaluation leader who facilitates, scribes who capture scenarios and proceedings, and questioners with expertise in particular qualities. The project decision makers are the architect, the project or product manager and the sponsor or customer who can act on the findings. The architecture stakeholders are everyone else with a stake: developers, testers, operations, security, integrators, support and user representatives. Independence matters: evaluators who designed the system, or who report to the architect, find fewer risks.
The phases of the ATAM
The nine steps
- 1. Present the ATAMThe evaluation leader explains the method, the participants' roles and the outputs, so that nobody mistakes the session for an audit or a design review of their personal work.
- 2. Present the business driversA senior business representative explains the system's goals, constraints, stakeholders and the qualities that matter most, in business terms. These business drivers are the yardstick for everything that follows.
- 3. Present the architectureThe architect describes the architecture at an appropriate level: context, main structures, runtime behaviour, deployment, and how it addresses the drivers. A typical time limit of about an hour forces focus.
- 4. Identify architectural approachesThe architect names the architectural approaches in use (styles, patterns and tactics such as a broker, replication or caching) and the evaluation team catalogues them, without yet analysing them.
- 5. Generate the quality attribute utility treeThe decision makers build or refine a utility tree and rate each leaf scenario for business importance and technical difficulty.
- 6. Analyse the architectural approachesFor the highest-priority scenarios, the evaluators ask which approaches address each one and probe them with quality-specific questions, recording risks, non-risks, sensitivity points and trade-off points.
- 7. Brainstorm and prioritise scenariosIn phase 2 the wider stakeholder group proposes scenarios of its own and votes on them. The result is compared with the utility tree; important scenarios missing from the tree are a finding in themselves.
- 8. Analyse the architectural approaches againThe highest-ranked new scenarios are analysed in the same way as in step 6, now with the knowledge of the stakeholders in the room.
- 9. Present the resultsThe evaluation team presents the approaches, scenarios, utility tree, risks, non-risks, sensitivity points, trade-off points and risk themes, and relates each theme to the business drivers it threatens.
The vocabulary of the analysis
The heart of the method is steps 6 and 8. For each important scenario the evaluators ask the architect to walk through how the architecture responds, and they probe with questions drawn from each quality: how many synchronous calls are on this path, what happens to in-flight requests when that node fails, where is this credential stored, what limits the queue? Each answer is classified using five concepts that have become standard vocabulary well beyond the ATAM itself.
The outputs of the analysis
| Output | Meaning | Example from a food-delivery platform |
|---|---|---|
| Risk | A decision, or a missing decision, that may prevent an important scenario or business goal from being met | Card authorisation sits inside the synchronous order path, so a slow payment provider slows every order |
| Non-risk | A decision judged sound for the scenario, recorded with the assumptions it depends on | The transactional outbox and idempotent consumers meet the no-lost-orders scenario, assuming the broker replicates each partition three times |
| Sensitivity point | A property of components or connections that is critical to one quality response | Order latency is sensitive to the number of synchronous hops and to the provider's p99 response time |
| Trade-off point | A property that is a sensitivity point for more than one quality, so that improving one worsens another | The courier location update interval: shorter improves map freshness but drains phone batteries and raises mobile data and ingestion cost |
| Risk theme | A group of risks with a shared underlying cause, linked to the business drivers it threatens | Third-party services sit on synchronous critical paths, threatening the goal of doubling order volume without losing conversions |
How the analysis produces its outputs
Non-risks deserve more attention than they usually get. Recording that a decision is sound because of a specific assumption (three replicas, a contractual response time from a supplier, traffic below a certain level) turns a vague reassurance into something that can be checked later. When the assumption changes, the non-risk becomes a candidate risk. Sensitivity and trade-off points are similarly valuable after the evaluation: they tell future maintainers which parameters to change with care, and they are natural places for the automated checks described in Section 11.3.
The utility tree revisited
Section 2.8 introduced the utility tree as a tool for prioritising quality requirements. It was born in the ATAM, and inside the method it plays two distinct roles. In step 5, the decision makers build the tree top-down: utility, then quality attributes, then refinements, then concrete scenarios rated (business importance, technical difficulty). The (H,H) leaves become the agenda for step 6, because they matter most and are hardest to achieve, so that is where risks are most likely to hide.
In step 7, the wider stakeholder group brainstorms scenarios bottom-up and votes on them, much as in a Quality Attribute Workshop. Comparing the two views is one of the most revealing moments of an evaluation. If the stakeholders' top scenarios already appear in the tree, the decision makers understand their system's purpose. If a scenario that receives many votes is absent, for example an operations scenario about restaurant tablets going offline, the decision makers have a blind spot, and the architecture may have one too. Such scenarios are added to the tree and analysed in step 8.
Worked example
Worked example: a condensed ATAM of the food-delivery platform
The platform from Chapters 2 and 4 connects customers, 4,000 restaurants and 9,000 couriers. The business drivers are to double order volume within 18 months (from 50 to 100 orders per second at the Friday peak) without losing conversions, and to sign national restaurant chains that demand contractual availability. The architecture: microservices per bounded context (ordering, restaurant, dispatch, notification) each with its own PostgreSQL database; an API gateway; synchronous REST for order placement, including card authorisation with an external payment provider; a Kafka broker with a transactional outbox for order events; courier positions sent every 4 seconds and held in Redis; one cloud region across three availability zones, with nightly backups copied to a second region. The phase 1 utility tree reused the six scenarios of Section 2.8.
| Order latency (H,H) | p95 under 1.5 s and p99 under 3 s at 100 orders/s. Approaches: gateway, four synchronous hops, card authorisation in the request. Measured provider latency: p95 900 ms, p99 2.5 s |
|---|---|
| Analysis | The provider alone consumes most of the p99 budget, leaving 500 ms for four hops and database writes. By Little's law (Section 2.4), 100 orders/s at 2.5 s means 250 requests in flight; the ordering service's pool of 200 workers saturates whenever the provider slows. Risk R1; sensitivity point S1: number of synchronous hops and provider p99 |
| Checkout uptime (H,H) | Payment provider times out at peak: orders accepted and queued, none lost, customer told within 10 s. Current design fails the order when authorisation times out. Risk R2 |
| Option raised | Authorise asynchronously: accept the order as pending, authorise from a queue. Improves latency and availability, but some accepted orders later fail payment and restaurants must not cook unpaid orders. Trade-off point T1: where in the order flow payment is confirmed |
| No lost orders (part of H,H) | Outbox in the same local transaction as the order, Kafka with three replicas, idempotent consumers. Non-risk N1, assuming replication factor 3 and producer acknowledgements from all in-sync replicas |
| Region failure (H,M) | RTO 15 min, RPO 1 min. Nightly backups give an RPO of up to 24 hours and a restore measured in hours. Risk R3 |
| Tracking freshness (M,M) | Map position never more than 5 s old. 9,000 couriers every 4 s is 2,250 writes/s, comfortable for Redis. Trade-off point T2: the update interval (freshness against battery, data and cost) |
| Account takeover (H,M) | Rate limiting by IP address at the gateway does not detect credential stuffing spread across thousands of addresses. Risk R4 |
| Phase 2 brainstorm | Restaurant and operations stakeholders voted highly for a scenario missing from the tree: a restaurant tablet offline for 10 minutes at peak must receive every order on reconnection. Push notifications are sent once with no acknowledgement. Risk R5, and a blind spot in the tree |
| Risk themes | Theme A (R1, R2, R4): third-party and edge dependencies sit on synchronous critical paths without isolation; threatens doubling volume without losing conversions. Theme B (R3, R5): recovery and delivery guarantees lag behind the availability being promised to restaurant chains; threatens the chain contracts |
| What happened next | Decision records for asynchronous authorisation with a pending state (accepting T1, with restaurants notified only after authorisation), a warm standby region, acknowledged order delivery to tablets, and credential-stuffing detection at the identity provider. N1's assumptions became an automated configuration check |
The findings are about decisions, not people or code quality, and each one is traceable to a scenario and a business driver.
Two of the five risks came from scenarios that the architect had not prioritised or not considered, which is why the stakeholder step exists.
Trade-off points are not problems to eliminate: T1 was accepted deliberately and recorded, with its consequences for restaurants and customers.
CBAM: putting money on the table
The ATAM tells you where the risks are; it does not tell you which fixes are worth paying for. The Cost Benefit Analysis Method (CBAM), developed at the SEI by Rick Kazman, Jai Asundi and Mark Klein (first published in 2001), builds on an ATAM's scenarios to make that economic case.
In outline, stakeholders take the prioritised scenarios and assign utility (on a 0 to 100 scale) to several response levels for each: worst case, current, desired and best case. The architects propose architectural strategies and estimate the response each would achieve. The utility gained by each strategy is read off the scenario's utility-response curve and weighted by the scenario's priority; summed across scenarios, that is the strategy's benefit. Dividing benefit by cost gives a value-for-cost ratio (VFC), and strategies are chosen in order of VFC within the available budget, then sanity-checked against intuition and constraints.
A miniature CBAM for the region-failure risk
| Strategy | Expected response | Utility gain x weight | Annual cost | Value for cost |
|---|---|---|---|---|
| Current: nightly backups | RTO about 6 h, RPO up to 24 h | utility 10 (baseline) | already paid | n/a |
| A: warm standby region, asynchronous database replication | RTO 10 min, RPO under 1 min | (90 - 10) x 0.3 = 24 | £180k | 24 / 180 = 0.13 |
| B: pilot light (data replicated, compute started on demand) | RTO 40 min, RPO under 1 min | (60 - 10) x 0.3 = 15 | £70k | 15 / 70 = 0.21 |
Strategy B gives more benefit per pound, but it misses the 15-minute recovery time that the restaurant contracts require. The numbers inform the decision; they do not make it. CBAM's real value is that it forces explicit, comparable estimates of benefit and cost, and exposes when a constraint, not a ratio, should decide. Its weakness is that utilities and costs are judgements; teams should vary them and see whether the ranking changes, exactly as Section 1.6 recommends for weighted decision matrices. Section 11.8 shows how to turn such estimates into a business case.
Costs and practicalities
A full ATAM is expensive in people's time. As a rough illustration: a four-person evaluation team spending two days each on preparation (8 person-days), a one-day phase 1 with three decision makers (7), a two-day phase 2 with ten stakeholders added (34) and a week of follow-up by two people (10) comes to around 60 person-days, before anyone fixes anything. For a system whose failure would cost millions, or that will run for a decade, that is cheap insurance. For a small product team it is out of proportion.
Several practices keep evaluations worthwhile. Scale the method: Bass, Clements and Kazman describe a lightweight architecture evaluation, a reduced ATAM for internal teams that can be completed in a day or less, and some organisations run only steps 2 to 6 with a handful of people. Prepare properly: most of the cost of a poor evaluation is spent discovering, in the room, that nobody can describe the architecture. Keep evaluators independent but informed. Agree beforehand who owns the findings and how they will be followed up, ideally as decision records and backlog items with owners. And repeat: the findings are a snapshot, and the risks change as the system and the business do.
Practitioners disagree about the method's place today. Its critics see it as a heavyweight artefact of large, up-front projects that fits poorly with teams deploying many times a day. Its defenders answer that the concepts (scenarios, sensitivity and trade-off points, risk themes) are exactly what agile teams lack when they skip evaluation, and that the method scales down. Both have a point: the full ceremony suits high-stakes, long-lived systems and major commitments, while the vocabulary and the habit of scenario-driven questioning belong everywhere. Section 11.2 covers the lighter end.
Questions people ask
Can the ATAM be applied to a system that already exists?
Yes. The steps are the same; the architecture is recovered from documentation, code and the people who run the system, and measurements of the live system replace estimates. Evaluating an existing system is common before a major extension, a migration or an acquisition.
Does a clean ATAM mean the system will meet its requirements?
No. The ATAM examines a sample of scenarios at the level of design decisions. It reduces the chance of unpleasant surprises but cannot replace measurement, testing and operational experience. Its findings should be turned into tests and fitness functions where possible.
Who should present the architecture?
The architect or lead engineer who knows it best. The evaluation team's job is to ask questions, not to present, and the architect's job is to explain and explore, not to defend.
Check your understanding
What is the difference between a sensitivity point and a trade-off point?
A sensitivity point is a property critical to one quality response. A trade-off point is a property that is a sensitivity point for several qualities at once, so that changing it helps one and hurts another.
Why does the ATAM record non-risks, and what must be recorded with them?
To show which decisions were examined and judged sound. The assumptions they depend on must be recorded too, so that the non-risk can be re-examined if an assumption changes.
What does it mean if stakeholders in phase 2 vote highly for a scenario that is missing from the utility tree?
The decision makers have a blind spot about what matters, and the architecture may not address that scenario. It is added to the tree and analysed in step 8.
In CBAM, how is a strategy's value for cost calculated?
Its benefit, the priority-weighted utility it adds across the scenarios it affects, is divided by its cost.
Sources and further reading
Have a question about this section? ArchitectBot, the course’s AI study assistant, answers from the course first.
Ask: “We are about to sign a three-year contract with a cloud provider for our payments platform. How would I run a reduced ATAM in two days to find the biggest architectural risks first?”