CI/CD pipelines and deployment strategies

From commit to production: what a pipeline is for

Section 0.8 (how software gets built) introduced continuous integration (every change is merged and verified automatically), continuous delivery (the main branch is always releasable) and continuous deployment (every change that passes the checks goes to production without a human decision). This section treats them as architecture: the design of the path a change takes, and the strategies that move it into production without hurting users.

The central idea is the deployment pipeline, described by Jez Humble and David Farley in Continuous Delivery (2010). Every commit triggers an automated sequence of stages. Early stages are fast and cheap and catch most problems; later stages are slower, more realistic and more expensive. A change moves to the next stage only if the previous stage passed, so the further a build gets, the more confidence it has earned. A failure anywhere stops that build and goes straight back to the person who made the change, while it is still fresh in their mind.

The pipeline matters to architects for three reasons. First, it is where deployability, a quality attribute from section 2.6, is actually achieved or lost. Second, its speed bounds how quickly the organisation can respond to a market change or a security vulnerability: if a one-line fix takes two days to reach production, the architecture is two days slow, whatever the code looks like. Third, the shape of the system constrains the shape of the pipeline: independently deployable services can have independent pipelines, but a distributed monolith forces coordinated releases however clever the tooling.

Build once, deploy many, and the artefact repository

A rule from Continuous Delivery that is still broken surprisingly often is build once, deploy many: compile and package each change exactly once, then promote that same artefact through every environment. If a team rebuilds for staging and again for production, the thing it tested is not the thing it shipped. A different compiler patch, a dependency resolved to a newer version overnight or a changed base image can make the two builds differ, and the tests prove nothing about the second one.

What differs between environments is configuration, supplied at deployment or start-up time (database addresses, credentials, feature-flag settings), never baked into the artefact. This is the configuration factor of the twelve-factor app. A container image tagged orders:4f1c2ab (the commit hash) is deployed to test, staging and production; only its environment variables and mounted secrets change.

The artefact repository is the store between build and deploy: a container registry (Amazon ECR, Azure Container Registry, Google Artifact Registry, GitHub Container Registry, Harbor) or a universal repository such as JFrog Artifactory or Sonatype Nexus for packages and binaries. Treat it as critical infrastructure. Deploy by immutable digest (sha256:...) rather than by a mutable tag such as latest, which can be moved to point at a different image; keep artefacts that are or were in production for as long as you might need to roll back to them or investigate them; and record provenance, meaning which commit, pipeline run and dependencies produced each artefact.

Trunk-based development and small batches

How code is branched decides how often it is really integrated. In trunk-based development everyone integrates small changes into one main branch (the trunk) at least daily, either by committing directly or through short-lived branches and pull requests that live for hours, not weeks. The DORA research programme has repeatedly associated short-lived branches and frequent merging with higher delivery performance, and it is a precondition for continuous integration in the literal sense: code on a branch for three weeks is not being integrated continuously.

Long-lived feature branches feel safer because unfinished work is kept apart, but the risk is merely postponed: the longer two branches diverge, the larger and riskier the eventual merge, and the longer the pipeline sees code that is not the code that will ship. GitFlow, a branching model with long-lived develop and release branches published by Vincent Driessen in 2010, was widely adopted; in 2020 Driessen added a note that teams delivering continuously, such as web applications, should consider a simpler workflow. Teams that do need to work on large changes in the trunk use two techniques: feature flags, which hide unfinished behaviour, and branch by abstraction, which introduces an interface in front of the old implementation, builds the new one behind it and switches over when ready.

Practitioners genuinely disagree at the edges. Open-source projects with many untrusted contributors rely on longer-lived pull requests and review, and regulated teams sometimes need explicit release branches for audit. The principle that survives the disagreement is the batch size: the smaller each change, the easier it is to review, test, deploy, diagnose and undo.

name: delivery
on:
  push:
    branches: [main]
env:
  IMAGE: registry.example.com/food/orders:${{ github.sha }}
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make lint test              # fast feedback first
      - run: docker build -t "$IMAGE" . && docker push "$IMAGE"
  staging:
    needs: build
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - run: ./deploy.sh staging "$IMAGE" && ./smoke-test.sh staging
  production:
    needs: staging
    runs-on: ubuntu-latest
    environment: production            # can require an approval
    steps:
      - uses: actions/checkout@v4
      - run: ./deploy.sh production "$IMAGE" --canary 5

Deployment strategies

A deployment strategy decides how the new version replaces the old one in production, and therefore how many users a bad version can hurt before it is stopped. The strategies differ in the extra infrastructure they need, how fast they roll back and whether two versions run at once. Whenever two versions run at once, they must be compatible with each other's data and messages, which is the price of every strategy except recreate.

Blue-green deployment was described by Martin Fowler in 2010, crediting Dan North and Jez Humble; the canary release, named after the canaries miners carried to detect gas, was written up by Danilo Sato on the same site in 2014. In Kubernetes, rolling updates are built in; blue-green and canary releases usually come from a progressive delivery controller (Argo Rollouts, Flagger) or a service mesh and the Gateway API's weighted routing. On managed platforms, the equivalent features are called traffic splitting or deployment slots.

Separating deployment from release with feature flags

Deployment installs a new version of the software in an environment; release makes a behaviour available to users. Treating them as one event means every deployment is a business event, with all the nervousness that brings. Feature flags (feature toggles) separate them: the new code is deployed switched off, and the behaviour is released later, gradually and reversibly, by changing configuration. Pete Hodgson's article Feature Toggles (aka Feature Flags) on martinfowler.com (2017) distinguishes four categories by how long they live and how dynamically they decide.

Flags have a carrying cost. Every flag doubles the paths through the code it guards, adds test combinations and leaves dead code once its decision is settled. Hodgson recommends treating flags as inventory: add a removal task when a release toggle is created, give flags expiry dates, add tests that fail once a flag is past its date, and limit how many may exist at once. The Knight Capital loss of 2012, described in section 1.3, involved a repurposed flag that reactivated old code on a server missed by a manual deployment: a reminder never to reuse flag names and to delete dead code behind settled flags. At scale, flags are served by a dedicated service (open-source options such as Unleash and Flagsmith, commercial ones such as LaunchDarkly, or the vendor-neutral OpenFeature API) so that a change propagates in seconds and is audited.

Database changes in the pipeline

Code can be rolled back in seconds; data cannot. A deployment that renames a column and is then rolled back leaves the old code facing a schema it does not understand. The answer, covered in depth in schema evolution, is to make every schema change backward-compatible with the version of the code currently running, using expand and contract: add the new structure first, deploy code that writes both and reads either, backfill, switch readers, and remove the old structure only in a later release once nothing uses it.

In the pipeline this means migrations are versioned artefacts that run as their own step, before the code that needs them is deployed, and are tested on a production-sized copy of the data so that a lock on a large table is discovered in staging rather than at lunchtime. Destructive steps (dropping columns or tables) are delayed and run separately, because they are the one kind of change a rollback cannot undo. A useful pipeline check fails the build when a migration is not additive unless it is explicitly marked as a contract step.

Progressive delivery and automated canary analysis

Progressive delivery, a term popularised by James Governor of the analyst firm RedMonk in 2018, combines canaries, feature flags and automated verification: a change reaches progressively larger audiences, and at each step the system, not a person watching dashboards, decides whether to continue. The decision uses automated canary analysis: the canary's metrics (error rate, latency percentiles, saturation, business signals such as orders placed) are compared statistically with a baseline. Netflix and Google open-sourced their analysis engine, Kayenta, in 2018; Argo Rollouts and Flagger provide similar analysis steps in Kubernetes.

Three details decide whether the analysis can be trusted. Compare like with like: a fresh baseline of the old version receiving the same share of traffic avoids blaming the new version for differences caused by cold caches or recent restarts. Have enough traffic: at 5% of a service handling ten requests a minute, an hour of canary sees thirty requests, too few to detect a 1% error rate, so low-traffic services need longer steps, synthetic load or a different strategy. Choose metrics tied to the service level objectives and to business outcomes, because a version can be fast and error-free while silently charging the wrong delivery fee.

Rollback or roll forward

When a release goes wrong, there are two ways out. Rollback returns to the previous artefact, which is fast and needs no new code, provided the previous version still works with the current data and with the versions of other services. Roll forward ships a fix through the pipeline, which is slower but sometimes the only option, for example after a data migration that cannot be reversed or a change that other services already depend on.

A sound default is roll back first, diagnose later: restore service with the fastest safe action, then investigate without pressure. Rolling forward under pressure tends to bypass the pipeline's checks, and a rushed fix to a production incident is a classic source of the next incident. Rollback is only a real option if it is designed in: previous artefacts kept, schema changes compatible in both directions for at least one version, messages readable by old and new consumers, and the rollback path itself exercised regularly. A team that has never rolled back should assume it cannot.

Measuring the pipeline

The DORA software delivery metrics introduced in section 2.6 (change lead time, deployment frequency, change fail rate, failed deployment recovery time and deployment rework rate) are largely properties of the pipeline and the architecture together. Small batches and trunk-based development shorten lead time and raise frequency; automated testing and canary analysis lower the change fail rate; fast rollback and flags shorten recovery time. Measure them per service, watch trends rather than single values, and never turn them into targets for individuals, which simply invites gaming.