Modularity: dividing a system into parts

What a module is

A module is a part of a system that has a clear responsibility and a defined way for other parts to use it. That definition is deliberately independent of size. In Section 0.3 a module was a Python file; in this chapter the word covers every unit of decomposition, the act of splitting a system into parts: a function, a class, a package, a library, a component inside one application, or a separately deployed service. The same questions apply at every scale: what is this part responsible for, what does it offer to others, what does it keep to itself, and what does it depend on?

Every module has two faces. Its interface is what other modules must know in order to use it: function names and parameters, an HTTP API, the events it publishes, and also the informal promises such as how fast it answers and what errors it can return. Its implementation is everything else: the code, the data structures, the database tables, the third-party libraries it happens to use. Good modularity keeps the interface small and stable and lets the implementation change freely.

David Parnas, whose 1972 paper is the foundation of this section, defined a module as a responsibility assignment rather than a piece of program text. That is the most useful way to think about it as an architect. A module is a promise that one part of the system, and usually one team, is responsible for one set of decisions. Whether the code ends up in one file or forty, one process or ten, is a separate question, discussed below under logical and physical modules.

Why modularise at all

A system of a few hundred lines can be understood as a whole. A system of a few million lines cannot, and nor can the organisation that builds it work as one undivided group. Modularity is the property of being divided into modules that can be understood, changed and reasoned about largely on their own. It is one of the five sub-characteristics of maintainability in ISO/IEC 25010 (Section 2.6), and it supports at least five distinct benefits.

Comprehension. A developer can study one module and its interfaces without reading the rest of the system. Parallel work. Teams can work on different modules at the same time with little need to coordinate, because they have agreed the interfaces in advance. Isolation of change. A change to one design decision, such as which tax rules apply, is made in one module and does not ripple outward. Reuse. A well-bounded module, such as an address-validation component, can be used by several products. Testing. A module with a clear interface can be tested on its own, with its dependencies replaced by test doubles.

Parnas listed three of these in 1972: managerial (development time shortened because separate groups work on separate modules with little communication), flexibility (drastic changes to one module without changing others) and comprehensibility (studying the system one module at a time). Half a century later, these are still the reasons. But notice that none of them follows automatically from cutting a system into pieces. A system can be divided into fifty modules and still deliver none of these benefits, if every change touches ten of them. The benefits depend entirely on where the boundaries are drawn.

Parnas and the KWIC example

Parnas made the point with a small program called KWIC (key word in context), which produces an index of titles. It reads lines of text, generates every circular shift of each line (moving the first word to the end, repeatedly), sorts all the shifts alphabetically and prints them. The line Design of Software produces Design of Software, of Software Design and Software Design of, so that a reader can find the title under any of its words.

The first decomposition he describes is the one most programmers would produce by drawing a flowchart: an Input module that reads the lines and stores them in memory in a packed format, a Circular Shift module that builds an index of where each shift starts, an Alphabetising module that sorts that index, an Output module that prints the result, and a master control module that calls them in order. It is a perfectly reasonable design. Each module is small and the interfaces between them are the data formats in memory.

The second decomposition looks stranger. It has a Line Storage module that offers functions such as get the character at line r, word w, position c and hides how lines are actually stored; an Input module that reads lines and hands them to Line Storage; a Circular Shifter that presents the shifts as if they were stored, through similar access functions; an Alphabetiser that offers give me the i-th shift in order without saying when or how it sorts; an Output module; and a master control. The modules no longer correspond to the steps in the processing.

Parnas then listed decisions that were questionable and likely to change: the input format; keeping all lines in memory, which is impractical for very large inputs; packing several characters into each machine word; storing an index of shifts rather than the shifts themselves; and sorting everything once rather than searching on demand. In the first decomposition most of these changes affect several modules, because every module knows the shared data formats. In the second, each change is confined to the one module that hides that decision. The second design was also easier to develop in parallel, because teams had to agree only function names and parameters, not detailed table layouts, and easier to understand one module at a time.

The criterion: decompose by decisions likely to change

Parnas's conclusion was that it is almost always incorrect to begin decomposing a system on the basis of a flowchart. Instead, he proposed starting from a list of difficult design decisions, or design decisions which are likely to change, and designing each module to hide one such decision from the others. He called the principle information hiding; Section 3.3 treats it in depth.

The idea sounds simple, but it reverses the instinct most of us have. A flowchart describes time: first this happens, then that. Design decisions cut across time. The decision about how an order is represented is used when the order is received, priced, paid for, stored and confirmed. If modules follow the steps, that one decision is spread across all of them, and changing it means changing them all. If a module owns the decision, the steps all ask that module, and the decision can change in one place.

Two refinements matter in practice. First, the criterion is about likely change, which means an architect has to make an informed guess about the future: what has changed in similar systems, what the business is planning, which suppliers and regulations are unstable. The four modifiability questions from Section 2.6 (what changes, how likely, who changes it, at what cost) are exactly the input this needs. Second, Parnas distinguished a good decomposition from a hierarchy. A system can be cleanly layered and still badly decomposed, and the reverse; Section 3.4 deals with the direction of dependencies separately.

Logical modules and physical modules

A logical module is a unit of responsibility in the design: Catalogue, Ordering, Payments. A physical module is a unit that is built, versioned or deployed: a Java JAR file, a Python wheel, an npm package, a container image, a serverless function. Section 1.1 called these module structures and allocation structures. The two are related but they are not the same thing, and confusing them causes some of the most expensive mistakes in modern architecture.

The same logical design can be mapped to physical modules in very different ways. All the logical modules can live in one deployable unit, calling each other in-process: a modular monolith. Or each can be deployed as its own service, calling the others over the network: microservices. The logical boundaries, and the decisions each module hides, can be identical in both. What changes is the cost of crossing a boundary (a function call takes nanoseconds, a network call milliseconds and can fail), the ability to release and scale parts independently, and the operational work.

A common and costly error is to treat physical separation as if it created logical modularity. Splitting a badly decomposed system into services does not hide any decisions; it turns in-process calls between tangled modules into network calls between tangled services, sometimes called a distributed monolith. The opposite error also occurs: a team keeps one deployable unit but lets its internal modules grow into each other, so that it can never later separate them. The practical advice from many practitioners is to get the logical boundaries right first, enforce them, and only then decide which of them deserve to become physical boundaries.

Granularity and its costs

How big should a module be? Granularity is the size of modules relative to the system, and there is no universally right answer, because the costs pull in opposite directions. Larger modules have fewer interfaces to design and maintain, but each contains more unrelated reasons to change, more people editing it at once and more to understand before making a change. Smaller modules are each easier to understand, but the system has more interfaces, more dependencies to manage, and more places where a single feature has to be stitched together.

The cost of an interface also depends on the kind of boundary. Between two classes in one program, an extra interface costs a little reading time. Between two services it costs a network hop, serialisation, a versioning policy, monitoring, a failure mode and possibly a second team's agreement to every change. That is why the right granularity for services is usually much coarser than the right granularity for classes.

Enforcing module boundaries in code

A boundary that exists only in a diagram will not survive contact with a deadline. Sooner or later someone will import an internal class because it is quicker, and once one shortcut exists others follow; Section 1.7 called this erosion. Languages and build tools offer several ways to make the intended boundary the path of least resistance, or to make crossing it fail the build.

Python relies on convention: names with a leading underscore are treated as internal (PEP 8), and a package's __init__.py can re-export only its public functions. Nothing stops an import of shop.payments._stripe_client, so teams that care use a checker such as import-linter to fail the build when a forbidden import appears. Java offers package-private visibility (the default when no modifier is given) and, since Java 9 in 2017, the Java Platform Module System: a module-info.java file declares which modules this one requires and which of its packages it exports; packages that are not exported cannot be used from outside, at compile time or at run time. Go treats any package under a directory named internal as importable only from code rooted at the parent of that directory. C# has internal, visible only within one assembly; Rust has pub(crate).

# shop/payments/__init__.py  (the public face of the package)
from shop.payments.api import charge, refund, PaymentResult

__all__ = ["charge", "refund", "PaymentResult"]

# shop/payments/_provider_client.py   internal: leading underscore
# shop/payments/_retry.py             internal
# shop/payments/api.py                the only module callers should use

# Elsewhere in the shop:
from shop.payments import charge          # fine
from shop.payments._retry import backoff  # works, but breaks the boundary

// module-info.java for the payments module
module shop.payments {
    requires shop.orders.api;        // what this module depends on
    exports shop.payments.api;       // the only package others can use
    // shop.payments.provider and shop.payments.retry are not exported,
    // so code in other modules cannot compile against them
}

Common mistakes

Decomposing by technical layer only. Folders called controllers, services and repositories are modules of a kind, but a change to how VAT works will touch all three. Layers are useful (Section 3.4), yet a decomposition that has only layers is a flowchart in disguise. Most mature systems combine a split by business capability with layering inside each part.

Shared everything. A module called common or utils that every other module imports becomes the place where unrelated decisions accumulate, and any change to it ripples everywhere. Boundaries without enforcement. Diagrams show clean modules while the code has hundreds of cross-imports. Modules defined by the org chart of a past reorganisation rather than by the decisions the system has to hide; Section 3.7 explains why organisations and module boundaries pull on each other. Premature distribution. Turning every logical module into a service before the boundaries have proved stable makes later boundary corrections expensive.

How professionals apply modularity

Experienced architects treat the module structure as the single most important design artefact for maintainability, and they revisit it. Early in a product's life, when the domain is poorly understood, they prefer fewer, larger modules with strong internal boundaries, because merging and splitting modules inside one codebase is cheap. As the domain stabilises and likely changes become clear, they refine the boundaries and only then consider which ones should become separately deployed. Techniques from domain-driven design, such as bounded contexts, help find boundaries that follow the language of the business.

They also measure. Counting the modules touched by recent changes, which version control history makes easy, shows whether the boundaries are working; Section 3.8 introduces change coupling and hotspots. The next section, Section 3.2, gives the vocabulary for describing what good and bad boundaries look like: coupling between modules and cohesion within them.