Writing quality attribute scenarios that can be tested

Why scenarios

Section 2.2 gave you the vocabulary of qualities; this section gives you the grammar. A quality name on its own, such as performance or security, cannot be designed for or tested. Even a number on its own is not enough: '2 seconds' means nothing until you know for which operation, under what load, measured where and for what share of requests. The quality attribute scenario, developed at the Software Engineering Institute (SEI) and set out by Len Bass, Paul Clements and Rick Kazman in Software Architecture in Practice, solves this by forcing every quality requirement into the same six-part shape.

The idea is simple. Every quality shows itself when something happens to a system: a surge of users, a crash, a change request, an attack. A scenario names that event, the circumstances, the part of the system affected, what should happen and how you will know it happened well enough. It is short (usually one to three sentences), concrete, and testable. And because every quality uses the same shape, scenarios for performance, security and modifiability can sit side by side in one prioritised list.

The six parts

The six parts are the source of stimulus, the stimulus, the artifact, the environment, the response and the response measure. Learn them in that order, since together they tell a short story: someone or something does something to part of the system under certain conditions; the system does something, and we measure it like this. The stimulus and the response measure are the two parts people most often leave out, and they are the two that matter most.

The environment deserves special care because the same stimulus can demand very different responses in different conditions. A crashed server during normal operation might require automatic failover within a minute; the same crash during a planned maintenance window might be acceptable with a manual recovery. A change request at design time (before release) costs far less than the same change once data has accumulated in production. Writing the environment explicitly stops people from comparing scenarios that are really about different situations.

The artifact matters for a similar reason. 'The system responds within 200 ms' hides which part is responsible. Naming the artifact (the search service, the API gateway, the mobile client) tells the team where the design work and the measurement must happen. For some qualities, such as modifiability and deployability, the artifact may be code, configuration or the delivery pipeline rather than a running component.

General and concrete scenarios

Bass, Clements and Kazman distinguish two kinds. A general scenario is system-independent: for one quality attribute it lists the typical values each part can take. For availability, for example, the sources might be internal or external, people, hardware, software or infrastructure; stimuli might be a crash, an omission, incorrect timing or an incorrect response; responses might be detecting, recording, recovering, masking or degrading; and measures might be time to detect, time to repair, availability percentage or the proportion of requests handled.

A concrete scenario picks specific values for one particular system: 'the primary order database crashes during normal operation; the replica takes over and writes resume within 60 seconds with no committed order lost'. General scenarios are templates and checklists; concrete scenarios are requirements. You use the general scenario to make sure you have considered the right sources, stimuli and measures, then write the handful of concrete scenarios that matter for your system.

From vague statements to testable scenarios

Most quality requirements start life as adjectives. The skill is to ask the questions that turn each adjective into a scenario: which operation? caused by whom? how often or how much? under what conditions? what should happen? what number tells us it worked? The table shows typical before-and-after pairs. Notice that the 'after' versions sometimes reveal that the stakeholder meant several different things, which then become several scenarios.

Scenario examples across six qualities

The examples below use one online shop so you can see how different qualities look in the same format. Each is written compactly with all six parts labelled; in practice teams often write them as one or two sentences.

Two features of these examples are worth noticing. First, scenarios are not only about running software. Modifiability and deployability scenarios have developers and product owners as sources, and their measures are effort, elapsed time and the scope of change. Second, good measures often combine a speed number with a correctness number: 'within 60 seconds' and 'no committed order lost'. A fast failover that loses data is not what the business wanted.

Common mistakes

Reviewing hundreds of scenarios written by teams reveals the same mistakes again and again. Each makes a scenario untestable or misleading.

From scenarios to acceptance tests and fitness functions

A well-written scenario is already most of a test. The source and stimulus tell you what load or event to generate, the environment tells you how to set up the test, the artifact tells you what to observe and the response measure gives the pass condition. A performance scenario becomes a load test; an availability scenario becomes a failover drill or a chaos experiment; a modifiability scenario becomes a review of how much code a sample change touched and how long it took; a security scenario becomes a penetration test case.

The next step is to keep checking. A fitness function, a term popularised by Neal Ford, Rebecca Parsons and Patrick Kua in Building Evolutionary Architectures, is any automated or repeatable check of how well the system meets an architectural goal. Run in the delivery pipeline or against production telemetry, it turns a scenario from a one-off acceptance test into a permanent guard: the build fails, or an alert fires, when the quality drifts. Chapter 11 covers fitness functions in depth in Fitness functions and evolutionary architecture.

# A performance scenario expressed as an automated check (illustrative)
# Scenario: at 300 searches/s, p95 <= 400 ms and p99 <= 1000 ms.
import statistics

def check_search_latency(latencies_ms, p95_target=400, p99_target=1000):
    cuts = statistics.quantiles(latencies_ms, n=100)  # 99 cut points
    p95, p99 = cuts[94], cuts[98]
    assert p95 <= p95_target, f"p95 {p95:.0f} ms exceeds {p95_target} ms"
    assert p99 <= p99_target, f"p99 {p99:.0f} ms exceeds {p99_target} ms"
    return p95, p99

# latencies_ms comes from a load test that replays 300 searches per second
# for ten minutes against a production-like environment.

How professionals use scenarios

In practice, teams rarely write long tables of six-part scenarios for everything. They write a small number, often ten to thirty for a substantial system, for the qualities that are architecturally significant, and they keep them where everyone can see them: in the architecture document (arc42 section 10 asks for exactly this), in the backlog as acceptance criteria, or in a shared quality board. Some teams use a shorter form; Gernot Starke's Q42 project, for instance, recommends a compact structure of context, trigger and acceptance criteria. The shape matters less than the discipline: every important quality has a stimulus, a context and a number.

Scenarios also power evaluation methods. The Quality Attribute Workshop and the Architecture Tradeoff Analysis Method (ATAM), both from the SEI, collect and prioritise scenarios with stakeholders and then test the architecture against the most important ones. Section 2.8 shows how scenarios are gathered and ranked in a utility tree.