Pipes and filters and pipeline architectures

Definition and origin

In the pipes and filters style, processing is a chain of independent steps. Each filter reads data from its input, does one job with it and writes the result to its output. Each pipe carries data one way from one filter to the next, usually with a buffer in between. Garlan and Shaw's description (1994) states the constraints that give the style its properties: filters must be independent, must not share state with other filters, and must not know the identity of the filters at either end of their pipes. A filter knows only the format of what it reads and what it writes.

The style's best-known home is Unix. In a memo dated 11 October 1964, Doug McIlroy wrote that there should be ways of coupling programs like a garden hose, screwing in another segment when the data needs to be massaged in another way. Ken Thompson implemented the pipe system call and pipes in the shell in Version 3 Unix in 1973, and the | notation followed in Version 4. The Unix philosophy that grew from it, write programs that do one thing well and work together through a universal text interface, is the pipes-and-filters style applied to a whole operating system's toolset. POSA (1996) catalogued Pipes and Filters as an architectural pattern, and Hohpe and Woolf's Enterprise Integration Patterns (2003) applied it to messaging between systems.

Richards and Ford call the same style the pipeline architecture, and they name four kinds of filter. A producer (or source) starts the flow, reading from a file, a database or a queue. A transformer changes each record: parsing, converting, enriching. A tester checks a condition and passes the record on or drops it, sometimes routing it to one of several outputs. A consumer (or sink) ends the flow, writing to a database, a file or a screen.

How it works

The constraints produce the benefits directly. Because a filter shares no state and does not know its neighbours, it can be written and tested alone, with a list of input records and an expected list of outputs; it can be reused in other pipelines; and filters can be added, removed or reordered by changing only how the pipeline is assembled. Because records flow incrementally, a filter can start work on the first record before the previous filter has finished the last, so stages run concurrently and the whole pipeline needs memory for only a few records at a time, not the whole data set.

The pipe hides how data moves. In a Unix shell it is an operating-system buffer between two processes; on Linux it holds 64 KiB by default, and a writer that fills it simply blocks until the reader catches up. In a Python program it can be a generator, so that each filter pulls the next record from the one before. In a data platform it can be a message queue or a topic in a log such as Apache Kafka, which lets filters run on different machines and survive restarts. The filter code barely changes between these; the operational properties change a great deal.

There are variations on the strict form. A pipeline can fan out (a tester routes records to two different consumers) or fan in (two producers feed one transformer). It can be linear, which is easiest to reason about, or a directed acyclic graph of steps, as in workflow tools such as Apache Airflow. Strictly, a filter with internal state, such as one that counts or sorts, breaks the incremental property, because it must see all input before producing output; sort in a Unix pipeline is such a filter. Such filters are allowed but they become the points where memory, latency and restarts need thought.

Examples: where pipelines appear

ETL and data pipelines. Extract, transform, load jobs move data from operational systems into a warehouse: extract yesterday's orders, clean and convert them, join them with restaurant data, load them into reporting tables. Each step is a filter, and modern data platforms (Apache Beam, Spark, dbt models, cloud data-flow services) are largely tools for building and running such pipelines at scale.

Compilers. A classic compiler is a pipeline of phases: lexical analysis turns characters into tokens, parsing turns tokens into a syntax tree, semantic analysis checks types and names, optimisation rewrites an intermediate representation, and code generation emits machine code. Each phase consumes one representation and produces the next, so phases can be developed and replaced separately; the LLVM project, for instance, lets many language front ends share one optimiser and set of back ends. Real compilers relax the strict style, because phases share a symbol table.

Media processing. Video transcoding and audio processing chain decoders, scalers, filters and encoders. GStreamer builds media applications explicitly as pipelines of elements linked through pads, and FFmpeg's filter graphs work the same way.

Log and event processing. Log shippers such as Logstash and Fluent Bit are configured as input, filter and output stages: read log lines, parse them into fields, drop health-check noise, mask personal data, and send them to a search index. The food-delivery platform's observability data flows this way.

Machine-learning feature and inference pipelines. Training data preparation, and the pre-processing and post-processing around a model, are commonly built as pipelines, which Chapter 12 returns to.

# Top ten URLs that returned HTTP 500 in today's access log
grep '" 500 ' access.log \
  | awk '{print $7}' \
  | sort \
  | uniq -c \
  | sort -rn \
  | head -10

import json

def read_lines(path):                 # producer
    with open(path) as f:
        for line in f:
            yield line

def parse(lines):                     # transformer
    for line in lines:
        yield json.loads(line)

def only_failed(events):              # tester
    for e in events:
        if e["status"] == "payment_failed":
            yield e

def count_by_provider(events):        # consumer
    counts = {}
    for e in events:
        counts[e["provider"]] = counts.get(e["provider"], 0) + 1
    return counts

events = only_failed(parse(read_lines("orders.jsonl")))
print(count_by_provider(events))

Batch and streaming

The same structure runs in two modes. Batch processing handles a bounded input, such as all of yesterday's orders, runs to completion and stops. It is simple to reason about: if a run fails, fix the problem and run it again, provided each run replaces its output rather than appending to it. Nightly payouts to restaurants, monthly invoices and warehouse loads are typical batch pipelines. Their weakness is delay: results are as old as the last run.

Stream processing handles an unbounded input, processing each record shortly after it arrives: courier positions, payment events, clicks. Results are fresh, but new questions appear. A stream never ends, so aggregates must be computed over windows (orders per restaurant per five minutes); records can arrive late or out of order; and the pipeline must run continuously, recover from crashes without losing or double-counting records, and cope when input rates spike. Frameworks such as Kafka Streams, Apache Flink and Apache Beam provide windows, checkpoints and state stores for these problems. Beam's model treats batch as a special case of streaming, with a bounded input, so one pipeline definition can run both ways.

Many organisations run both: a stream for fresh, approximate operational dashboards and a batch job that recomputes exact figures overnight. Section 7.8 discusses the data-architecture side of this choice.

Back pressure and error handling

When a producer is faster than a consumer, something has to give. The options are to buffer (which only postpones the problem unless the buffer is bounded), to drop or sample records (acceptable for some metrics, never for payments), or to apply back pressure: make the fast stage slow down to the speed of the slow one. Unix pipes do this automatically because a full pipe blocks the writer. Pull-based designs such as Python generators do it by construction, because nothing is produced until the next stage asks for it. Distributed pipelines need it designed in: bounded queues, consumers that pull at their own pace from a log, or protocols such as the JVM's Reactive Streams specification in which a subscriber signals how many records it is ready for. Section 9.3 covers queues and back pressure in depth.

Errors need a policy for each filter, decided up front. A malformed record (a poison message) should not stop the pipeline: route it to a dead-letter queue or error file with the reason, count it, and alert if the rate rises. A transient failure, such as a lookup service timing out, is retried with back-off. A crash of the pipeline needs a restart point: batch jobs rerun from the start or from a checkpoint, and streaming frameworks checkpoint their position and state. Because restarts mean some records are processed twice, consumers that write results should be idempotent, for example by upserting on a natural key rather than inserting blindly. Errors are where the style is weakest: filters are deliberately unaware of each other, so a failure discovered at step four cannot easily undo what step two already did, and a pipeline that needs that kind of coordinated rollback is probably the wrong style.

When to use it, when not to, and anti-patterns

Use pipes and filters when the problem is naturally a one-way flow of data through distinct transformations: ingestion, conversion, enrichment, validation, aggregation, export. It is the default for data engineering, for processing files and media, and for any job where the steps change more often than the overall flow. It also appears inside other styles: a nightly reporting job within a modular monolith, or a stream-processing service within an event-driven system.

Avoid it as the main structure of interactive applications, where users expect immediate, conversational responses and the work is not a flow; for logic that needs many steps to share and update the same state; and for processes that need coordinated rollback across steps, which call for a transaction or a saga.

Common anti-patterns: the god filter, one step that parses, enriches, validates and writes, which loses the benefits of the style; hidden shared state, filters that communicate through a shared database table or global variable instead of the pipe, which reintroduces the coupling the style removes; format drift, a producer that silently changes its output format and breaks every downstream filter, prevented by explicit schemas and schema checks (see Section 7.7); unbounded buffers, which turn a slow consumer into an out-of-memory crash; and non-idempotent sinks, which double-count after every retry.

Enforcing the style in code

The style's constraints can be made explicit. Give every filter the same simple signature (an iterable of records in, an iterable out), so that composition is mechanical. Define the record format at each pipe as a schema (a dataclass, a JSON Schema, an Avro or Protocol Buffers definition) and validate it at the boundary. Keep filters free of shared mutable state: an architecture test can forbid filter modules from importing the database layer or each other, so that the only way data passes between them is the pipe. Test each filter with small input and output fixtures, and test the assembled pipeline end to end with a representative sample, including malformed records, to prove the error policy works.

In distributed pipelines, monitor per-stage throughput, queue depth and consumer lag: a growing queue in front of one stage is the visible sign of a bottleneck or a failing filter, and it is the first metric to alert on.