Platform engineering and internal developer platforms

The problem platforms solve

In 2006 Amazon's chief technology officer, Werner Vogels, summed up his company's operating model as you build it, you run it: the team that writes a service also deploys, monitors and fixes it. The idea became central to DevOps and to microservices, and it works: teams that run their own software feel the consequences of their design choices and fix them. But the cloud-native toolset described in this chapter is large. A product team that must build, run and secure a service today may need to understand container images, Kubernetes objects, infrastructure as code, pipelines, observability, secrets management, network policies, supply-chain scanning and cost controls, before writing a line of business logic.

When every team solves all of this alone, three things happen. Effort is duplicated, as ten teams write ten slightly different pipelines. Quality varies, as some teams encrypt their databases and configure backups and others forget. And the time product teams spend on infrastructure is time not spent on the product. Platform engineering is the response: a dedicated team builds an internal developer platform (IDP) that packages these capabilities so that product teams can use them through self-service.

Evan Bottcher's 2018 article on martinfowler.com gives a widely quoted description: a digital platform is a foundation of self-service APIs, tools, services, knowledge and support, arranged as a compelling internal product. Each part of that description matters. Self-service means no tickets for routine needs. Knowledge and support means documentation and people, not only software. Compelling means teams choose to use it because it is better than doing the work themselves. Internal product means it is managed like a product, with customers, rather than like a project or a cost centre.

An IDP is not one product. It is usually an integration of existing tools (a CI system, a GitOps controller, a cloud provider's managed services, an observability stack, a secret store) behind consistent interfaces and sensible defaults. The platform team's distinctive work is choosing, integrating, documenting and supporting them, and designing the experience of using them, not writing a new orchestrator.

Golden paths and paved roads

The most valuable thing a young platform can offer is a golden path: an opinionated, supported and documented way to accomplish a common task end to end, such as creating a new service and getting it into production. Spotify popularised the term (its engineering blog described its golden paths in 2020); Netflix speaks of the paved road. The metaphor is precise. A paved road is the easiest route, and it is maintained, but you may still drive off-road if you have a good reason; you just do not get the road's support.

A good golden path has five properties. It is end to end: it takes a team from an empty repository to a monitored service in production, not just through one step. It is opinionated: it makes the common choices (language versions, logging format, deployment strategy, database defaults) so that teams do not have to. It is optional: teams with unusual needs can leave it, which keeps the platform team honest about whether the path is actually better. It is supported: when it breaks, the platform team fixes it. And it is evolving: services created from last year's template receive updates, rather than being frozen at the version they started with.

Backstage and developer portals

A developer portal is the front door of many platforms. The best-known is Backstage, built inside Spotify, open-sourced in March 2020, accepted by the Cloud Native Computing Foundation (CNCF) as a sandbox project in September 2020 and promoted to incubation in March 2022. Its core features are a software catalogue, in which every service, website, library, API and resource is registered with its owner and relationships; software templates, which generate new projects from golden-path templates; TechDocs, which publishes documentation written as Markdown next to the code; and a large ecosystem of plugins that show pipelines, Kubernetes status, costs and incidents alongside each catalogue entry.

Each component describes itself in a small file kept in its own repository, so ownership information lives with the code and is reviewed like code:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: restaurant-ratings
  description: Stores and serves customer ratings of restaurants
  tags: [python, http]
spec:
  type: service
  lifecycle: production
  owner: team-restaurants
  system: restaurant-discovery
  providesApis: [ratings-api]
  dependsOn: [resource:ratings-db]

Two cautions. Backstage is a framework that an organisation assembles and operates, typically needing engineers who know TypeScript and React, and its plugins vary in quality; managed Backstage services and commercial portals exist for teams that do not want to run it. More importantly, a portal is not a platform. A beautiful portal in front of manual, ticket-driven processes merely displays the queue more attractively. Start with the capabilities and the golden path; add a portal when there is enough behind it to be worth finding.

The platform as a product

The single most important idea in platform engineering is that the platform is a product and the developers who use it are its customers. That means a product manager or a platform lead who owns the roadmap; user research (interviews, observing a team create a service, surveys) before building; documentation and onboarding treated as features; published service level objectives for the platform itself; versioning and deprecation policies so that teams can plan; and, above all, adoption that is earned rather than mandated. If teams avoid the platform, that is market feedback, not insubordination.

Team Topologies (Matthew Skelton and Manuel Pais, 2019), covered in section 11.5, gives the organisational model. A platform team exists to reduce the load on stream-aligned teams, those aligned to a flow of business change such as ordering or courier dispatch, and interacts with them mainly in X-as-a-Service mode: a clear, self-service offering with little need for collaboration. Collaboration mode is used deliberately and briefly, for example when designing a new capability with one or two early-adopter teams. Skelton and Pais also coined the thinnest viable platform: the smallest platform that genuinely helps, which for a small organisation may be a well-written wiki page of approved services and a shared pipeline template.

This also means the platform must obey Conway's law in its favour: the platform's interfaces become the boundaries between the platform team and every product team, so they should be stable, documented and narrow, like any good abstraction.

Self-service with guardrails

Self-service removes the queue; guardrails keep it safe. The difference from traditional controls is where the check happens. A gate is a person who must approve a request, such as a change advisory board or a database administrator reviewing a ticket. A guardrail is an automatic limit that lets anything through that is within policy and blocks or flags what is not, at the moment of the request. Guardrails are usually written as policy as code (introduced in section 10.4) and enforced in pipelines, admission controllers and cloud organisation policies.

Good guardrails share three properties: they give a clear message explaining what was blocked and how to fix it; they are versioned and tested like code, with a period of warnings before a new rule starts blocking; and they come with a documented route for justified exceptions, because a guardrail with no exception path turns back into a gate the first time it is wrong.

Reducing cognitive load, not hiding everything

Team Topologies borrows the idea of cognitive load from John Sweller's educational research and distinguishes three kinds. Intrinsic load is inherent in the task, such as understanding how delivery fees work. Extraneous load comes from the environment, such as remembering the twelve steps to request a database. Germane load is the effort of learning that makes future work easier, such as understanding the domain more deeply. A platform should strip out extraneous load so that teams can spend their limited attention on intrinsic and germane load in their own domain.

That does not mean hiding everything. A team that runs its service still needs to understand how it behaves in production: its resource use, its failure modes, its latency and its costs. Abstractions that hide too much leak at the worst moment, during an incident, when the team discovers it cannot see the logs or does not know which database tier it is on. Good platforms are transparent: they provide sensible defaults and simple interfaces while letting teams look underneath and override when they need to.

Measuring whether the platform works

A platform's value shows up in other teams' outcomes, so its measures must look there. Useful measures fall into four groups, and a balanced set takes at least one from each.

The SPACE framework, published in ACM Queue in 2021 by Nicole Forsgren, Margaret-Anne Storey and colleagues from GitHub, Microsoft Research and the University of Victoria, warns that developer productivity cannot be captured by one number. It names five dimensions: satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. It recommends measuring several dimensions at once, including at least one perceptual measure such as a survey, and never judging individuals by activity counts such as commits or lines of code.

Evidence on platforms is encouraging but not unconditional. The 2024 DORA Accelerate State of DevOps report found that users of internal developer platforms reported about 8% higher individual productivity and 10% higher team performance, yet software delivery throughput and change stability were 8% and 14% lower. The report offers possible explanations, such as extra hand-offs and the platform layer itself adding steps, and the lesson for architects is to measure delivery outcomes as well as satisfaction, and to expect a dip while a platform matures.

Risks and failure modes

Platform initiatives fail in recognisable ways. The commonest is building a platform nobody wants: a team spends a year building an ambitious general-purpose platform, launches it, and finds product teams already have workable solutions and no time to migrate. Starting thin, with real users and one golden path, is the antidote. The second is the renamed operations team: an infrastructure team is relabelled as a platform team but still works from a ticket queue, so nothing changes except the job titles. If routine requests still need a ticket and a person, it is not yet a platform.

Other risks are mandating adoption, which removes the feedback that keeps a platform useful; over-abstraction, in which a home-grown layer hides the underlying tools so completely that teams cannot debug or extend it and the platform team becomes a bottleneck for every change; neglecting the long tail of services created from old templates; and the platform becoming a single point of failure whose outage stops every deployment. Conversely, a small organisation may need no platform team at all: a few shared templates maintained by volunteers can be the thinnest viable platform for years.