Observability: logs, metrics and traces

Monitoring and observability

Monitoring means collecting predefined measurements and checks and raising an alarm when they cross a threshold: disk 90% full, error rate above 1%, health check failing. It answers questions you knew to ask in advance. Observability is a property of the system: how well you can understand its internal state, including states nobody predicted, from the telemetry it already emits, without shipping new code to find out. The word comes from control theory, where Rudolf Kálmán defined it in 1960 as the ability to infer a system's internal state from its outputs.

The distinction matters because distributed systems mostly fail in new ways. A food-delivery order can be slow because one restaurant's tablet app retries aggressively, because one payment provider's endpoint in one region is degraded, or because a new menu feature makes one query slow only for restaurants with more than 500 items. No one wrote a dashboard for any of these. Monitoring tells you that checkout is slow; observability lets you ask for whom, where and why by slicing the telemetry by any attribute until the pattern appears.

Practitioners disagree about how far the distinction goes. One view, common among tool vendors, treats observability as an umbrella over three pillars: logs, metrics and traces. Charity Majors, Liz Fong-Jones and George Miranda (Observability Engineering, O'Reilly, 2022) argue that the pillars framing misses the point, and that observability requires rich, high-cardinality, structured events that can be queried in arbitrary combinations. Both camps agree on the architectural consequence: observability is a quality attribute that must be designed in, through consistent instrumentation, shared identifiers and context propagation, because it cannot be bolted on during an incident.

Structured logging

A log line such as Payment failed for order 81F3 after 3001ms is easy to write and hard to use: every query needs a fragile regular expression, and every developer phrases things differently. Structured logging writes each entry as key-value fields, usually JSON, with agreed names: timestamp, level, service, event name, business identifiers (order, restaurant, courier), outcome, duration and the trace and span identifiers that link the line to its trace. Queries become precise (all card_authorisation_failed events for provider cardco in the last hour, grouped by http_status), and logs from different services can be joined.

A few rules keep logs useful and affordable. Log events, not prose; use a fixed event name per kind of event. Use levels consistently, and do not log at error level for conditions nobody needs to act on. Never log secrets, tokens, full card numbers or unnecessary personal data; telemetry pipelines are copied, shipped to third parties and kept for months, so they are a common source of data leaks, and personal data in logs falls under the GDPR like any other copy (Section 8.7). Security logging and detection, a related but different purpose with its own retention and integrity needs, is covered in Section 8.8.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             version-trace id (16 bytes)-parent span id (8 bytes)-flags (01 = sampled)

{"ts": "2026-10-22T12:31:07.412Z", "level": "error",
 "service": "payment-service", "event": "card_authorisation_failed",
 "order_id": "ord_81F3", "provider": "cardco", "http_status": 504,
 "duration_ms": 3001, "attempt": 2,
 "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7"}

Metrics and why percentiles cannot be averaged

Most metric systems, such as Prometheus (a CNCF graduated project) and the OpenTelemetry metrics model, offer the same few types. A counter only increases, such as total requests served; you read it as a rate (requests per second over the last five minutes), which also survives restarts. A gauge is a current value that rises and falls: queue length, connections in use, memory. A histogram counts observations into buckets (for example requests under 50 ms, under 100 ms, under 250 ms and so on) plus a total count and sum, from which percentiles are estimated. Prometheus also has a summary type that computes quantiles inside each process.

The distinction between histograms and summaries is not a detail. As Section 2.4 noted, a percentile is a property of a whole distribution, and the percentile of a combined distribution cannot be computed from the percentiles of its parts. Averaging p99 values across servers, or across minutes, produces a number that is not a percentile of anything. Bucket counts, by contrast, can simply be added: sum each bucket across all servers, then estimate the percentile from the merged histogram. That is why fleet-wide latency SLIs should be built on histograms (or on counts of requests under a threshold, as in Section 9.5), never on per-instance summaries.

What to measure: RED, USE and the golden signals

Three short checklists cover most of what a service should expose. The RED method, proposed by Tom Wilkie in 2015, applies to anything that serves requests: Rate (requests per second), Errors (failed requests per second) and Duration (the distribution of response times). The USE method, from Brendan Gregg, applies to resources: for every CPU, disk, network link, connection pool and queue, check Utilisation (how busy it is), Saturation (how much work is waiting) and Errors. The four golden signals from the Google SRE book's chapter on monitoring distributed systems are latency, traffic, errors and saturation: in effect RED plus saturation.

In practice, RED describes what users experience and USE explains why. If the payment service's duration rises (RED), USE on its resources might show a connection pool at 100% utilisation with 40 requests waiting (saturation), which in turn points to the slow card provider holding connections open. Measure latency separately for successes and failures: a fast error is not good latency, and a timeout is both an error and a slow response.

Distributed tracing

In a monolith a stack trace shows where a slow request spent its time. Across services there is no single stack. Distributed tracing rebuilds one: every request gets a trace identifier at the edge, every unit of work becomes a span (with a name, start time, duration, status, attributes and the identifier of its parent span), and a tracing backend assembles the spans into a tree. Google's Dapper paper (Sigelman and colleagues, 2010) described the approach at scale: low overhead, instrumentation in shared libraries rather than in application code, and sampling so that only a fraction of requests is recorded.

Tracing depends entirely on context propagation: each service must pass the trace context to the next. The W3C Trace Context specification (a W3C Recommendation since 2020) standardises this for HTTP with a traceparent header holding a version, a 16-byte trace ID, the 8-byte ID of the calling span and flags such as whether the trace is sampled, plus an optional tracestate header for vendor data. For messaging, the same values travel in message headers, and the consumer's span refers back to the producer's. A single service that drops the header (an old library, a hand-written HTTP client, a batch job) breaks every trace that passes through it, which is why propagation belongs in shared platform libraries, not in each team's code.

Recording every span of every request is expensive at scale, so traces are sampled. Head-based sampling decides at the start of the request (keep 1 in 100) and is cheap, but it throws away most of the rare slow and failed requests you care about. Tail-based sampling buffers spans until the request finishes and then keeps all errors, all requests slower than a threshold and a small fraction of the rest; it needs more memory in the collector but keeps the interesting traces.

OpenTelemetry: one standard for instrumentation

Until about 2019, each tracing and monitoring vendor had its own agents and libraries, so changing vendor meant re-instrumenting every service. OpenTelemetry (OTel) was formed in 2019 by merging two earlier open-source projects, OpenTracing and OpenCensus, under the Cloud Native Computing Foundation (CNCF), and graduated as a CNCF project in May 2026. It provides language APIs and SDKs, automatic instrumentation for common frameworks and clients, the OTLP wire protocol, the Collector, and semantic conventions that fix attribute names such as http.request.method and http.response.status_code so that telemetry from different teams and languages lines up. Traces, metrics and logs are stable; profiles were added as a fourth signal and entered public alpha in 2026.

The architectural value is the separation of instrumentation from backend. Services emit OTLP to a Collector, usually running beside each node or as a central gateway; the Collector batches, samples, removes sensitive attributes, adds metadata such as the deployment version, and exports to whichever stores the organisation uses. Changing vendor, or sending the same data to two backends during a migration, becomes a Collector configuration change instead of a code change in two hundred services. A platform team typically provides the SDK setup, propagation and Collector as a paved road, so that product teams only add business attributes.

from opentelemetry import trace
import structlog

tracer = trace.get_tracer("order-service")
log = structlog.get_logger()

def place_order(order):
    with tracer.start_as_current_span("place_order") as span:
        span.set_attribute("restaurant.id", order.restaurant_id)
        span.set_attribute("order.item_count", len(order.items))
        quote = pricing.quote(order)       # instrumented HTTP client adds
        payment.authorise(order, quote)    # the traceparent header itself
        log.info("order_placed", order_id=order.id,
                 total_pence=quote.total_pence)
        return order.id

Cardinality and cost

Telemetry can easily cost more than the system it observes. The main driver for metrics is cardinality: every distinct combination of label values is a separate time series that must be stored and indexed. Labels with a few values (HTTP method, status class, region) are fine. Labels with unbounded values (user ID, order ID, full URL with identifiers in it, error message text) multiply series without limit and can bring a metrics system down.

Other cost levers are sampling traces (tail-based sampling keeps the useful ones), shorter retention for detailed data and longer retention for aggregates, dropping debug logs in production by default, and charging telemetry back to the teams that produce it so that its cost is visible. Treat telemetry volume as a design parameter with a budget, like any other resource.

Alerting on symptoms, not causes

The SRE book's guidance is that pages should be about symptoms users feel, not about the many possible causes. High CPU, a full disk on one replica or a restarted container are causes: often harmless, sometimes already handled by automation, and never a complete list of what can go wrong. A burning SLO, failing checkouts or a stalled courier-position pipeline are symptoms: they matter whatever their cause. Symptom-based alerts are fewer, rarely false and cover failures nobody predicted; cause-based signals belong on dashboards that help diagnose a symptom once it is paged.

Every page should be urgent, actionable and real. Alert fatigue is a reliability risk in itself: on-call engineers who are woken by meaningless alerts learn to dismiss them and miss the one that matters. A practical rule is to page on SLO burn rates (Section 9.5), send slower burns and resource warnings to a ticket queue, and link every page to a runbook and a dashboard.

Dashboards that answer questions

A wall of fifty graphs answers no question quickly. Good dashboards are designed like any user interface: for a named audience and a specific question. A service overview answers is this service healthy? with its SLOs, remaining error budget and RED metrics at the top, and the USE metrics of its key resources below. A journey dashboard for the on-sale answers are fans getting tickets? with admissions, seat holds, payments and failures per minute end to end. Diagnostic dashboards are reached by drilling down, not shown first.

Keep dashboards in version control as code, give each an owner, show deployments and feature-flag changes as annotations on the graphs (most incidents follow a change), and delete dashboards nobody has opened for months. During an incident, the first useful view is usually the user-facing journey, the second the trace of a failing request, and only then the resource metrics.