Encryption, key management and secrets

What encryption protects, and what it does not

Encryption is often treated as a box to tick: "data is encrypted in transit and at rest". The phrase hides the question that matters, which is against whom. Every layer of encryption protects data from some parties and not from others, and in every case the protection is exactly as strong as the control over who can use the key. Encryption does not remove the access-control problem; it moves it from the data to the keys, where it is usually easier to manage, audit and revoke.

Encryption in transit protects data moving across a network from being read or altered on the way. On the web that means TLS, introduced in Chapter 0; between services it means TLS or mutual TLS, discussed with zero trust. Encryption at rest protects stored data, but at rest is not one thing: the table shows the common layers on the food-delivery platform.

Reading down the table, each layer protects against more parties and costs more functionality. A sound design uses the cheap, transparent layers everywhere (TLS and storage encryption are close to free today) and the expensive ones selectively, for the data whose exposure would do the most harm. Deciding which data that is is the job of data classification, covered later in this section together with tokenisation and a worked example.

Symmetric and asymmetric cryptography

Symmetric encryption uses one secret key to encrypt and decrypt. It is fast enough for bulk data, and the standard algorithm is AES (FIPS 197, 2001) with 128- or 256-bit keys. Modern designs use authenticated encryption, which detects tampering as well as hiding content: AES in Galois/Counter Mode (AES-GCM) or ChaCha20-Poly1305 (RFC 8439). Authenticated modes need a unique nonce for every message encrypted under the same key; reusing a nonce with AES-GCM can reveal plaintext and allow forgeries, which is one of many reasons to use well-tested libraries rather than assembling primitives yourself.

Asymmetric cryptography uses a key pair: a public key that anyone may hold and a private key kept secret. It enables two things symmetric cryptography cannot. Key agreement lets two parties who have never met establish a shared symmetric key over an open network (elliptic-curve Diffie-Hellman in TLS 1.3). Digital signatures (RSA, ECDSA, Ed25519) let anyone with the public key verify that data came from the private key's holder and was not changed, which gives integrity, authenticity and a basis for non-repudiation. Asymmetric operations are slow, so real protocols are hybrid: TLS uses asymmetric cryptography to authenticate the server and agree keys, then symmetric encryption for the data.

A large quantum computer would break today's RSA and elliptic-curve algorithms, though not AES with sufficiently long keys. In August 2024 NIST published its first post-quantum standards: FIPS 203 (ML-KEM, for key establishment), FIPS 204 (ML-DSA) and FIPS 205 (SLH-DSA), both for signatures. Data that must stay confidential for many years is at risk today from attackers who record encrypted traffic now to decrypt it later, so the architectural response is crypto agility: keep an inventory of where which algorithms are used, avoid hard-coding them, and prefer libraries, protocols and providers that can switch, as major browsers and TLS libraries have begun to do with hybrid post-quantum key exchange.

Hashing, MACs, signatures and encryption are different tools

Confusing these primitives is a common source of design flaws, such as "encrypting" passwords so that they can be decrypted, or trusting a plain hash to prove that a webhook came from a payment provider. The table sets them side by side.

Storing passwords

A general-purpose hash such as SHA-256 is designed to be fast, which is exactly wrong for passwords: graphics processors can test billions of guesses per second against a stolen table of fast hashes. Password hashing uses functions that are deliberately slow and, ideally, memory-hard, so that each guess costs an attacker real time and memory, with a unique random salt per password so that identical passwords produce different hashes and precomputed tables are useless.

The OWASP Password Storage Cheat Sheet gives current recommendations in order of preference. Argon2id (a variant of Argon2, winner of the 2015 Password Hashing Competition), with a minimum of 19 MiB of memory, two iterations and one degree of parallelism (or an equivalent trade-off, such as 46 MiB with one iteration). scrypt where Argon2id is unavailable, with a cost parameter of 2¹⁷ (128 MiB), a block size of 8 and parallelism of 1. bcrypt only for legacy systems, with a work factor of at least 10, remembering that most implementations ignore input beyond 72 bytes. Where FIPS 140 compliance is required, PBKDF2 with HMAC-SHA-256 and 600,000 iterations. These figures rise as hardware improves, so check the current cheat sheet rather than copying numbers into a design document.

Three design points follow. Store the algorithm and parameters with each hash (the standard encoded formats do this), so that parameters can be raised later: when a user logs in successfully, verify against the old hash and immediately rehash with the new parameters. Consider a pepper, a secret added to every password before hashing and kept in a vault or HSM rather than the database, which defeats an attacker who steals only the database. Most important, ask whether you should store passwords at all: delegating authentication to an identity provider and offering passkeys removes the most attacked secret from most of your services.

Key management and envelope encryption

Keys have a lifecycle, described in NIST SP 800-57 Part 1: generation (from a proper random source), storage, distribution, use, rotation at the end of a defined cryptoperiod, revocation if compromised, and destruction. Doing all of that safely by hand is impractical, which is why architectures rely on a key management service (KMS) such as AWS KMS, Azure Key Vault or Google Cloud KMS, or on a self-run system such as HashiCorp Vault. A KMS holds master keys inside hardware security modules (HSMs), tamper-resistant devices validated against FIPS 140, and performs operations with those keys on request, subject to an access policy, so that applications never see the master keys and every use is logged.

Sending every record to the KMS to be encrypted would be slow and expensive, and KMS APIs limit request sizes. Envelope encryption solves this. The application asks the KMS for a fresh data encryption key (DEK); the KMS returns it twice, in plaintext and wrapped (encrypted) under a key-encryption key (KEK) that never leaves the HSM. The application encrypts the data locally with the plaintext DEK, discards the DEK from memory and stores the ciphertext with the wrapped DEK. To decrypt, it sends only the small wrapped DEK to the KMS, which checks the caller's permission, logs the request and returns the DEK.

import os
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

def encrypt_field(plaintext: bytes, kms, kek_id: str, context: bytes) -> dict:
    dek = AESGCM.generate_key(bit_length=256)            # fresh data key
    nonce = os.urandom(12)                               # never reuse with this key
    ct = AESGCM(dek).encrypt(nonce, plaintext, context)  # context binds it to its row
    wrapped = kms.encrypt(key_id=kek_id, plaintext=dek)  # KEK stays inside the KMS
    return {"ct": ct, "nonce": nonce, "wrapped_dek": wrapped, "kek": kek_id}

def decrypt_field(rec: dict, kms, context: bytes) -> bytes:
    dek = kms.decrypt(key_id=rec["kek"], ciphertext=rec["wrapped_dek"])  # checked, logged
    return AESGCM(dek).decrypt(rec["nonce"], rec["ct"], context)

Envelope encryption has several architectural benefits. Rotation is cheap: rotating a KEK creates a new key version for new wrapping operations, while old versions remain available for decryption; existing DEKs can be re-wrapped without touching the data, and full re-encryption is needed only if a DEK itself may have been exposed. Blast radius is limited: with a DEK per record, file or partition, one leaked DEK exposes little. Separation of duties becomes possible: the ordering service may use its KEK to unwrap but not change its policy or delete it, while security administrators can manage keys but cannot decrypt data. Key hierarchies map onto the organisation: a KEK per service and data class, or per tenant, so that a large restaurant chain could even supply and control its own key. And destroying a key is a powerful tool: crypto-shredding makes every copy of the data encrypted under it unreadable, including backups and immutable event logs, which matters for erasure under the GDPR and for event-sourced systems.

The price is a new critical dependency. If the KMS is unavailable, encrypted data cannot be read; if a key is deleted by mistake, the data is gone for good. Cloud KMSs therefore enforce a waiting period before deletion, and architects should treat key deletion and key policy changes as privileged, logged, two-person operations, cache unwrapped DEKs briefly where the latency of KMS calls matters, and include the KMS in disaster recovery plans.

Secrets management

A secret is any credential that grants access: database passwords, API keys for the payment provider, private keys, signing keys, OAuth client secrets, webhook secrets. Secrets management starts from one rule: secrets never live in source code, nor in container images, nor in configuration files committed to repositories, because those are copied, cached, forked and read by many more people and systems than the secret should be. Once in a repository's history, a secret stays there after the line is deleted.

Secrets belong in a vault (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager) that stores them encrypted, releases them only to authenticated workloads according to policy, logs every access and supports rotation. Better than rotating long-lived secrets is not having them: a vault can issue dynamic secrets, such as a database account created for one service instance with a lease of an hour, and workload identity lets services authenticate to cloud APIs and to each other without any stored secret at all. Secrets injected at run time should be read from files or the vault's API rather than broadly visible environment variables, which tend to end up in crash dumps and debug logs.

Prevention needs automation, because people make mistakes. Pre-commit hooks and pipeline scanners (such as gitleaks and detect-secrets) catch secrets before they are pushed; repository hosts scan for known secret formats, and GitHub's push protection blocks pushes that contain them. Public repositories are scanned by attackers within minutes of a push. The consequences are well documented: in 2016 attackers who gained access to a private GitHub repository used by Uber engineers found an access key for Uber's cloud storage there and downloaded personal data of about 57 million riders and drivers.

Tokenisation and reducing PCI DSS scope

Tokenisation replaces a sensitive value with a token that has no mathematical relationship to it; the real value is kept in a separate, tightly protected vault, and only the vault can map the token back. Unlike encryption, a stolen token cannot be reversed with a key, because there is no key, only a lookup in a system the attacker does not control.

Its most important use is card payments. The Payment Card Industry Data Security Standard (PCI DSS, currently version 4.0.1, published in June 2024) applies to every system component that stores, processes or transmits cardholder data, or that is connected to or could affect the security of those components. A platform that lets card numbers pass through its servers brings its API gateway, services, logs, networks and administrators into scope, with hundreds of requirements and an annual assessment. A platform that uses the payment provider's hosted payment fields or redirect, so that card numbers go straight from the customer's device to the provider, receives only a token and perhaps the last four digits, and its PCI DSS scope shrinks dramatically: a smaller merchant may qualify for the shortest self-assessment questionnaire, and even the largest merchants, which still need an annual assessment by a qualified assessor, have far fewer systems to assess. This is the design in the checkout threat model: the cheapest card data to protect is card data you never hold. Encrypting card numbers yourself does not achieve the same reduction, because whoever can decrypt them remains in scope.

The same idea works for other sensitive identifiers. An internal tokenisation service can replace national insurance numbers or passport numbers with tokens throughout the platform, so that only one small, heavily protected service ever sees the real values.

Data classification

Encryption, tokenisation, logging, retention and access decisions all depend on knowing which data is how sensitive. A data classification scheme gives a few levels with handling rules attached, so that teams do not decide each case from scratch. Four levels are common; what matters is that every data set in the platform's data catalogue or schema carries a level, and that the rules are enforced by templates and checks rather than by memory.

Field-level encryption

Field-level encryption, the application-level layer from the first table, encrypts individual fields before they are written, typically with envelope encryption and a KEK per data class. Its strength is that everything downstream of the application holds only ciphertext: database administrators, read replicas, backups and the data warehouse fed by change data capture. A breach of any of them exposes nothing readable, and access to the plaintext is reduced to the services that can call the KMS with that key, each call logged.

Its costs are real. The database can no longer index, search, sort or join on the encrypted field. Equality lookups can be restored with a blind index, a separate column holding a keyed hash (HMAC) of the normalised value, at the price of revealing which rows share a value. Deterministic encryption has the same leak, while range queries need specialised schemes or should simply not be done on such fields. Some databases offer client-side field-level or queryable encryption, which packages these techniques. Because of these costs, field-level encryption should be reserved for restricted data, whose fields are rarely queried anyway.