Containers and orchestration with Kubernetes

Images, layers and registries

Chapter 0 introduced containers as isolated processes that share the host's kernel, packaged as images built from a Dockerfile (servers and the cloud). An architect needs a little more detail, because image design affects build speed, security and how reliably the same software reaches every environment.

A container image is a stack of read-only layers, each a compressed set of file changes, plus a small configuration document (the command to run, environment variables, the user) and a manifest that lists them. Every layer and manifest is identified by a digest, a SHA-256 hash of its content. Identical layers are stored and downloaded once, so a hundred images built on the same base share its layers. Because each instruction in a Dockerfile produces a layer and builders reuse cached layers until something changes, instructions are ordered from least to most frequently changed: base image, then dependencies, then application code. Multi-stage builds compile in one stage and copy only the result into a small final image, leaving compilers and build tools behind.

Images live in a registry: Docker Hub, GitHub Container Registry, Amazon ECR, Azure Container Registry, Google Artifact Registry, or a self-hosted one such as Harbor. A tag such as orders:2.4.1 is a movable label; anyone with push rights can point it at a different image tomorrow. A digest such as orders@sha256:3f1c... can never change meaning. Production deployments should therefore resolve tags to digests, so that what was tested is exactly what runs, and should never use latest.

Small images are faster to pull and have fewer components to patch. Common practice is a minimal base (a slim distribution or a distroless image containing only the runtime), running as a non-root user, rebuilding regularly to pick up security fixes, and scanning images for known vulnerabilities in the pipeline. Signing images and attaching a software bill of materials belong to supply-chain security, covered in Chapter 8 (software supply-chain security).

The OCI standards

Docker popularised containers from 2013, and in 2015 Docker and others founded the Open Container Initiative (OCI) under the Linux Foundation to stop the format fragmenting. The OCI maintains three specifications. The runtime specification says how to run an unpacked container (namespaces, control groups, mounts, the process to start); runc is its reference implementation. The image specification defines layers, manifests and configuration. The distribution specification defines the HTTP API that registries expose for pushing and pulling. Version 1.1 of the image and distribution specifications (2024) added standard ways to store other artefacts in registries and to link them to an image, which is how signatures and software bills of materials now travel alongside the images they describe.

The practical result is interchangeability. An image built by Docker, Podman, Buildah or a cloud build service can be stored in any compliant registry and run by any compliant runtime: containerd and CRI-O on a Kubernetes node, a serverless container platform or a developer's laptop. That is why containers, unlike most packaging formats before them, became a lingua franca between development, operations and every major cloud.

Why orchestration is needed

Running one container is a single command. Running the food-delivery platform is not: forty services, three to thirty copies each, spread over dozens of machines in three availability zones, with a new version of something going out every hour. Someone has to decide which machine each copy runs on, restart copies that crash, replace machines that die, add copies for the evening rush and remove them afterwards, roll out new versions without dropping requests, let services find each other as copies come and go, hand them configuration and secrets, and attach storage to the few that need it.

Container orchestration is software that does all of this from a declaration of what should be running. Google ran an internal system called Borg for this for more than a decade; in 2014 it announced Kubernetes, an open-source system drawing on that experience, released version 1.0 in July 2015 and donated it to the newly founded Cloud Native Computing Foundation, where in 2018 it became the first project to graduate. Alternatives exist (HashiCorp Nomad, Amazon ECS, serverless container platforms), but Kubernetes is now the common substrate offered by every major provider and by most on-premises platforms.

How Kubernetes works: control plane and nodes

A Kubernetes cluster has a control plane, which makes decisions, and worker nodes, which run the workloads. The API server is the front door: every client, from a developer's kubectl to the cluster's own components, reads and writes objects through its REST API, and it authenticates the caller, checks authorisation and runs admission controls that can validate or modify each request (for example, rejecting containers that run as root). It is the only component that talks to etcd, a strongly consistent key-value store that replicates the cluster state with the Raft consensus protocol across, typically, three or five members, so that a majority survives the loss of one or two (consistency, CAP and PACELC).

The scheduler watches for new pods with no node assigned, filters out nodes that cannot run them (not enough unreserved CPU or memory, wrong zone, missing hardware, taints the pod does not tolerate), scores the rest (spreading copies across zones, packing efficiently) and records its choice through the API server. The controller manager runs dozens of controllers, each responsible for one kind of object: the ReplicaSet controller keeps the right number of pods, the Node controller notices nodes that stop reporting, the Job controller runs tasks to completion. In a cloud, a cloud controller manager also creates load balancers and attaches disks through the provider's API.

Each worker node runs a kubelet, an agent that watches the API server for pods assigned to its node, asks the container runtime (usually containerd or CRI-O, through the Container Runtime Interface; Docker Engine's built-in shim was removed in version 1.24 in 2022) to start them, runs their health probes and reports status back. kube-proxy programs the node's packet-forwarding rules so that traffic sent to a Service's virtual address reaches one of its pods; some network plugins, such as Cilium, replace it with an eBPF data plane. A network plugin gives every pod its own IP address, and a cluster DNS service (usually CoreDNS) lets pods find Services by name.

Notice the direction of the arrows. The control plane never logs in to a node to run a command; nodes watch for the state they should be in and converge on it. This pull-based, watch-and-converge design is what makes Kubernetes robust: a node that was disconnected for ten minutes simply catches up when it returns. With managed Kubernetes (Amazon EKS, Azure Kubernetes Service, Google Kubernetes Engine and others) the provider runs and upgrades the control plane; the customer still chooses node types, upgrades node pools, and owns everything deployed into the cluster.

Declarative desired state and reconciliation loops

The central idea of Kubernetes is declarative configuration. You do not tell the cluster to start three containers; you store an object saying that three copies of a given image should exist, the desired state. Controllers then run reconciliation loops: observe the actual state, compare it with the desired state and act to remove the difference, forever.

Follow what happens when a node in zone B fails during the evening rush. The Node controller stops receiving heartbeats and marks the node not ready; after a timeout its pods are marked for eviction. The ReplicaSet controller for the orders service observes two running pods where three are desired and creates a new pod object. The scheduler sees an unscheduled pod and assigns it to a healthy node, preferring zone B's survivors or another zone according to the spreading rules. That node's kubelet sees the assignment, pulls the image and starts the container. When its readiness probe passes, the endpoints behind the orders Service are updated and traffic flows to it. No component told any other what to do; each reacted to the state it observed.

Because controllers are level-triggered (they react to the current difference) rather than edge-triggered (reacting to individual events), a controller that crashes, misses a notification or runs twice still converges to the right result. The same pattern can be extended: custom resource definitions add new object types, and the operator pattern, introduced by CoreOS in 2016, packages a controller that encodes operational knowledge, for example creating, backing up and failing over a PostgreSQL cluster from a single declared object. Section 10.4 shows the same loop applied to whole environments through GitOps.

Declarative control has a cost: you must think in terms of end states and let go of exact sequencing. When two controllers want different desired states for the same field, they fight. A classic example is a Deployment manifest in Git that says replicas: 3 while an autoscaler sets it to nine; every GitOps sync scales the service back down in the middle of the rush. The fix is to let exactly one owner manage each field, here by removing replicas from the manifest.

A Deployment and a Service in YAML

The manifests below declare the orders service: configuration from a Secret, resource requests and probes, a stable Service in front and an autoscaler that keeps between three and twelve replicas. The Deployment deliberately has no replicas field, so that the autoscaler is the only owner of the count. They are applied with kubectl apply or, better, by a pipeline or GitOps agent from a repository.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
spec:
  # no replicas field: the autoscaler below owns the count
  selector:
    matchLabels: {app: orders}
  template:
    metadata:
      labels: {app: orders}
    spec:
      containers:
      - name: orders
        image: registry.example.com/orders:2.4.1
        ports: [{containerPort: 8080}]
        env:
        - name: DATABASE_URL
          valueFrom: {secretKeyRef: {name: orders-db, key: url}}
        resources:
          requests: {cpu: 500m, memory: 512Mi}
          limits: {memory: 512Mi}  # no CPU limit: may burst
        readinessProbe:
          httpGet: {path: /ready, port: 8080}
          periodSeconds: 5
        livenessProbe:
          httpGet: {path: /live, port: 8080}
          periodSeconds: 10
          failureThreshold: 3

apiVersion: v1
kind: Service
metadata:
  name: orders
spec:
  selector: {app: orders}
  ports: [{port: 80, targetPort: 8080}]
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: orders}
  minReplicas: 3
  maxReplicas: 12
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: {type: Utilization, averageUtilization: 60}

Resource requests, limits and quality of service

Requests and limits are the most consequential lines in a manifest. A request is what the scheduler reserves: a pod requesting 500 millicores (half a CPU) and 512 MiB of memory is placed only on a node with that much unreserved capacity, and it is guaranteed that share under contention. A limit is a ceiling enforced by the kernel: a container that exceeds its CPU limit is throttled (slowed down), and one that exceeds its memory limit is killed and restarted, the familiar OOMKilled status. Kubernetes derives a quality-of-service class from these settings: Guaranteed (requests equal limits for every resource), Burstable (requests set, limits higher or absent) and BestEffort (nothing set, first to be evicted when a node runs short).

Practitioners disagree about CPU limits. One camp sets them everywhere for predictability and fairness between teams; another sets accurate CPU requests and no CPU limits, because throttling causes latency spikes even when the node has idle CPU, and requests already guarantee each pod its share. For memory there is little debate: set the limit equal to the request, because memory cannot be reclaimed from a process without killing it, and a pod using more than it requested is the first candidate for eviction. Either way, requests must be based on measurements; inflated requests waste money (the node is reserved but idle), and understated ones cause noisy-neighbour problems and misleading autoscaling.

The HorizontalPodAutoscaler computes desired replicas = ceiling(current replicas × current metric ÷ target), measuring CPU use as a percentage of the request. It scales up quickly but, by default, waits for a five-minute stabilisation window before scaling down, to avoid flapping. When new pods cannot fit on existing nodes they stay Pending, and a node autoscaler (the Cluster Autoscaler, or Karpenter on some providers) adds machines, which takes minutes rather than seconds. Pod autoscaling and node autoscaling are separate loops with separate delays, and both must be planned for the rush.

Liveness, readiness and startup probes

Probes are how Kubernetes learns about health. A readiness probe answers whether this pod should receive traffic now; while it fails, the pod is removed from its Service's endpoints but left running. A liveness probe answers whether the process is stuck beyond recovery; when it fails repeatedly, the kubelet restarts the container. A startup probe holds off the other two until a slow-starting application has finished initialising.

The most damaging mistake is a liveness probe that checks dependencies. If the orders service's liveness endpoint queries the database, a thirty-second database failover makes every orders pod fail liveness at once, Kubernetes restarts them all, they all reconnect at the same moment, and a brief blip becomes an outage. Liveness should check only that the process itself can respond; readiness may consider whether the pod can do useful work, though even there many teams avoid making all pods unready because of a shared dependency, preferring to fail requests fast with a circuit breaker (see resilience patterns).

Probes work together with graceful shutdown. When a pod is deleted during a rollout or scale-down, Kubernetes removes it from endpoints and sends the process SIGTERM; the process should stop accepting new work, finish requests in flight and exit within the grace period (30 seconds by default) before it is killed. Because endpoint removal propagates asynchronously, a short preStop delay avoids sending requests to a pod that has already begun to stop. Section 10.3 returns to this as the twelve-factor principle of disposability.

When Kubernetes is overkill

Kubernetes is a general-purpose platform for building platforms, and it charges for that generality. Someone must design networking, ingress, identity, secrets, policies, observability and multi-tenancy; keep up with three minor releases a year, each supported for roughly fourteen months, plus the add-ons that depend on them; and be on call for the cluster as well as the applications. Managed control planes remove some of this work, not most of it.

For a small team running a handful of stateless web services, a PaaS or a serverless container platform (Google Cloud Run, Azure Container Apps, Amazon ECS with Fargate and similar) gives most of the benefits (containers, autoscaling, rolling deployments, managed TLS) with a fraction of the operational load. Kubernetes starts to pay when there are many services and several teams, a platform team to run it, workloads with unusual needs (stateful systems, GPUs, batch jobs, custom controllers), a requirement to run the same platform on premises and in several clouds, or an ecosystem of tools the organisation wants to use. Many organisations have adopted it because it was fashionable and then spent years building, in effect, a worse PaaS on top of it. Section 10.7 discusses how platform teams avoid that fate (platform engineering).