Why architecture matters: cost of change, quality, risk and real failures

The decisions that are expensive to get wrong

Section 1.1 defined architecture as the set of structures and significant decisions that shape a system, with significance measured by cost of change. This section asks the practical follow-up: so what? Why should a team, a product owner or a finance director care about architecture, rather than simply about features?

The short answer is that the architecture decides, more than any other technical factor, what the system will be like: how fast, how reliable, how secure, how cheap to run, and above all how easy to change. Features are what a system does. Architecture determines how well it does them and what it will cost to do something different next year. Many large software failures and many expensive successes trace back to a handful of structural decisions.

Architecture determines quality attributes

A quality attribute is a measurable property of how well a system does its job: performance, availability, security, modifiability, scalability, usability, cost-efficiency and others. Chapter 2 makes these precise. The key point here is that most quality attributes are emergent: they arise from how the parts are arranged and how they interact, not from any one piece of code.

Consider availability. If an online shop's checkout must make five synchronous calls in sequence to services that are each available 99.9% of the time, the chain as a whole can be no more available than roughly 99.5%, however well each service is written, because the probabilities multiply. Adding a queue so that some of those steps happen later, or caching results so that some calls are not needed, changes the arithmetic. Neither change is visible inside any single function; both are architectural. Performance, security and cost behave in the same way. A secure login page does not make a system secure if an internal service with no authentication is reachable from the internet.

Functionality, by contrast, is usually largely independent of structure. You can implement the same features of a banking app as one deployable unit or as fifty services; users see the same buttons. What changes is everything else: how quickly a fix can be released, what happens when one part fails, how many engineers the system can absorb, and what the cloud bill looks like.

Cost and speed of change

Most of a successful system's cost is incurred after its first release, as it is changed to meet new needs. The structure therefore matters most because it governs how expensive each future change will be. In a well-structured system, a change in tax rules touches one module. In a tangled one, it touches twenty, each needing testing and coordination.

Martin Fowler's design stamina hypothesis (2007) captures the effect. A team that pays little attention to internal structure often delivers faster at first. Over time, the accumulated tangle slows every change, and a team that invested in structure overtakes it. Fowler calls this a hypothesis because it is hard to measure rigorously, and he suggests the crossover usually comes within weeks rather than months. The slowdown is the interest on technical debt, which Section 1.7 covers.

Risk, teams, communication and estimation

Risk. Architecture is where most technical risks are either contained or amplified. A design with a single point of failure, an untested integration between independently built parts, or an unproven technology at its core carries risk that no amount of careful coding removes. Much architectural work is risk management: identifying what could go badly wrong and shaping the structure so that it either cannot happen or does limited damage when it does.

Splitting work between teams. Melvin Conway observed in 1968 that organisations which design systems tend to produce designs that mirror their own communication structures. This is now called Conway's law, and it works in both directions. A system whose parts have clear boundaries and narrow interfaces lets several teams work in parallel with little coordination. A system where everything depends on everything forces every team into every meeting. Architecture and organisation design are therefore intertwined, as Chapter 3's section on Conway's law explains.

Communication. An architecture gives people a shared vocabulary and a shared map. New engineers learn the system faster, product managers understand why some requests are cheap and others are expensive, and operations staff know what depends on what during an incident.

Estimation and planning. Decomposition into known parts makes estimation possible: a change can be traced to the components it touches, and each can be estimated by the team that knows it. Without a clear structure, estimates are guesses, because nobody can say with confidence how far a change will ripple.

Case study: Knight Capital, 1 August 2012

Knight Capital was one of the largest market makers in US equities. According to the US Securities and Exchange Commission's order of 16 October 2013, in the first 45 minutes of trading on 1 August 2012 Knight's automated order router (called SMARS) sent millions of unintended orders to the market while processing 212 small retail orders. It received more than 4 million executions in 154 stocks, over 397 million shares, and was left with large unwanted positions. Knight lost more than USD 460 million, needed emergency outside investment within days, and the SEC later fined it USD 12 million for violating market-access rules.

The SEC order describes the mechanism. Years earlier, Knight had stopped using an old function called Power Peg, but its code remained in the router and could still be triggered: it was dead code that was not dead. In 2012 Knight wrote new code for a new NYSE programme, the Retail Liquidity Program, and that new code reused a flag that had formerly switched on Power Peg. The new code was deployed manually to the router's eight servers. A technician did not copy it to one of them, no second technician reviewed the deployment, and Knight had no written procedure requiring such a review. When the flag was switched on, seven servers ran the new code; the eighth interpreted the flag as "run Power Peg" and sent orders without recognising that they had already been filled. Automated emails warning of a problem were sent before the market opened but were not acted on.

It would be unfair to call this simply a coding bug, and equally unfair to call it purely an operations error. The architectural lessons are about structure and deployability: obsolete functionality left reachable in a production system; one flag given two meanings in different versions of the code; a deployment process with no automated verification that all servers ran the same version; and, as the SEC emphasised, inadequate controls to limit the damage once something went wrong. Each is a decision about how the system is built and released, not about a single line of code.

Case study: the HealthCare.gov launch, October 2013

HealthCare.gov, the US federal health insurance marketplace, opened for enrolment on 1 October 2013. Many people could not create accounts or complete applications, and the site suffered outages for weeks. The US Government Accountability Office (GAO) and the Department of Health and Human Services' Office of Inspector General (OIG) later investigated, and their findings are the best source for what went wrong.

The GAO's July 2014 report (GAO-14-694) found that the Centers for Medicare and Medicaid Services (CMS) issued contracts before key requirements were settled, that requirements kept changing, that oversight was weak, and that CMS launched the system without verifying that it met its performance requirements. Obligations on the main marketplace contract rose from USD 56 million to more than USD 209 million between 2011 and early 2014. A second GAO report in March 2015 (GAO-15-238) found that the systems had not been fully tested before launch, and that problems at launch included inadequate capacity planning, software coding errors and planned functions that had not been implemented. The OIG's 2016 case study identified the absence of clear leadership and poor coordination between policy and technical work as central causes. After intensive corrective work, the site was able to serve high consumer demand within about two months.

The HealthCare.gov story is mostly about programme management, but its architectural lesson is direct. The marketplace was built by several contractors as separate parts that had to work together, including a data hub connecting to other federal agencies, and nobody verified end to end, under realistic load, that the assembled whole met its capacity and performance goals. Quality attributes such as capacity are properties of the whole system, so they can only be assured by someone who owns the whole system's structure and tests it as a whole.

Case study: Prime Video and architecture as fit, not fashion

In March 2023, an engineer on Amazon Prime Video's Video Quality Analysis team published a post on the Prime Video Tech blog about a service that monitors video streams for quality defects. The first version had been built from distributed serverless components: AWS Step Functions orchestrated the steps and AWS Lambda functions did the work, with intermediate video frames passed between steps through Amazon S3 storage. The team reported that this design hit a hard scaling limit at around 5% of the expected load and would have been too expensive at full scale, largely because of the per-step orchestration charges and the many storage requests.

The team moved the media conversion and defect-detection steps into a single process running in containers on Amazon ECS, passing frames in memory instead of through storage, and scaled by running more copies of that process. The post reported that infrastructure costs fell by over 90% and that the service could scale further. The post was widely shared as "Amazon abandons microservices", which it did not say: the change concerned one service, and the author noted that microservices and serverless components do work at high scale, but that the choice between them and a monolith has to be made case by case.

The lesson is not that monoliths beat serverless. It is that architecture is about fit. The serverless design was quick to build and appropriate for an early, smaller workload. When the workload's defining characteristic (very high volumes of data passed between steps for every second of every stream) became dominant, a structure that moved that data across the network many times was the wrong fit, and a different structure fitted better. A good architect asks what the workload, the team and the budget need, not what is fashionable.