Security in operation: logging, detection and response

Security does not end at deployment

The earlier sections of this chapter aim to prevent attacks. No prevention is perfect, so mature organisations also design on the assumption that a breach will happen: they want to notice it quickly, limit its damage, recover cleanly and learn from it. Chapter 2 described the security tactics as resisting, detecting, reacting to and recovering from attacks, and showed how to write security scenarios with measurable responses such as a leaked credential is detected and revoked within 15 minutes. This section is about the architecture and practices that make such responses achievable.

Much of it builds on the operational foundations of observability and incident management in Chapter 9. The difference is the adversary: a failing disk does not hide its tracks, delete its logs or come back next week through a door it left open.

Security logging: what to log and what never to log

Security logs exist to answer, after the fact, who did what, to which resource, when, from where and with what result. The OWASP Logging Cheat Sheet is the standard practitioner reference. It recommends recording events such as authentication successes and failures, authorisation failures, session management failures, input validation failures, application errors, high-risk functions such as administrative actions and data exports, and the use of security controls themselves. Each event should carry when (a synchronised timestamp), where (application, service, host, environment), who (the authenticated identity, the source address) and what (the action, the object, the result and the reason).

Equally important is what must stay out. Logs are copied widely, kept for a long time and read by many people and tools, so they are a favourite place for secrets and personal data to leak from. Never log passwords, session identifiers or access tokens (log a hash or a key ID if you need to correlate), encryption keys and connection strings, payment card data or special-category personal data. Treat log data as personal data where it identifies people, and apply the minimisation and retention rules of Section 8.7. Redaction is best enforced centrally in the log pipeline as well as in code, because one forgotten debug statement can undo it.

{
  "ts": "2026-10-21T19:42:07.318Z",
  "event": "authz.denied",
  "service": "orders",
  "env": "prod",
  "actor": {"type": "user", "id": "cust_4411", "session": "sha256:7c1e..."},
  "client": {"ip": "203.0.113.24", "ua_family": "iOS app 7.3"},
  "action": "GET /orders/81723",
  "resource_owner": "cust_9902",
  "reason": "not_owner",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

Logs must also be protected from attackers, whose first step after gaining access is often to erase the evidence. Ship security logs promptly to a separate account or service that the application's own credentials cannot modify; store them with write-once retention; restrict and log access to them; and keep clocks synchronised so events from different systems line up. Treat log content as untrusted input: an attacker who can put a newline into a logged field can forge fake entries (log injection), so encode or reject control characters, and remember that Log4Shell was triggered by logging attacker-controlled text.

Audit trails

An audit trail is a narrower, stronger kind of log: a complete, chronological, tamper-evident record of defined actions, kept to establish accountability. Debug logs are sampled, rotated and changed whenever developers like; an audit trail must be complete for its scope (every refund, every role change, every access to a customer's support record), immutable, attributable to an authenticated identity and retained for a defined period. Regulations and standards often require one; PCI DSS, for example, requires audit log history to be retained for at least twelve months, with the most recent three months immediately available for analysis.

Architecturally, design the audit trail as a product of the domain, not a by-product of logging. Emit audit events from the service that owns the action, at the point the action commits (the transactional outbox guarantees both happen or neither does), into an append-only store with restricted write access. Tamper evidence can be added by hash-chaining, where each record includes a hash of the previous one, or by periodically anchoring a signed digest elsewhere. Event-sourced systems get much of this for free, as Section 7.5 showed. Cloud providers' control-plane audit logs, which record every API call made to the cloud account, are part of the same picture and must be enabled in every account and region.

Detection: SIEM, rules and response automation

A security information and event management (SIEM) system collects security telemetry from across the estate, normalises it, correlates events and raises alerts when patterns match detection rules: many failed logins followed by a success, a service account used from a new country, a role granted administrator rights outside a change window, a large export at three in the morning. Typical sources are the API gateway, application security events, the identity provider, cloud audit logs, Kubernetes audit logs and endpoint detection and response (EDR) agents on staff laptops and servers.

Good detection is engineered, not bought. Derive rules from the threat model and from MITRE ATT&CK, a public knowledge base of real attackers' tactics and techniques maintained by MITRE since 2015; write them as code in a shared format such as Sigma, version them, and test them by simulating the attack they should catch. Measure alert quality: a rule that fires fifty times a day and is always benign trains people to ignore it, and alert fatigue is how real attacks are missed. Cheap, high-signal tripwires help: honeytokens, such as a fake access key or a fake customer record that no legitimate process ever touches, produce almost no false alarms. Security orchestration, automation and response (SOAR) tools then run playbooks for well-understood alerts, such as disabling a user, revoking a key or isolating a container, so that people handle the judgement calls.

SIEM ingestion is often priced by volume, which tempts teams to drop logs. A tiered design helps: high-value security events go to the SIEM for real-time detection; bulky logs go to cheaper write-once storage where they can be searched during an investigation.

Vulnerability management

New vulnerabilities in the software you run are published every day. A CVE (Common Vulnerabilities and Exposures) identifier, such as CVE-2021-44228, gives each publicly known vulnerability a unique name; the programme, started by MITRE in 1999, assigns identifiers through hundreds of CVE Numbering Authorities, including major vendors and open-source foundations. The US National Vulnerability Database adds analysis such as affected product identifiers and scores, although a backlog in that enrichment from 2024, and a brief scare in April 2025 when the contract funding the CVE programme nearly lapsed, showed the risk of depending on a single feed; most organisations now combine several sources, including GitHub's advisory database and the open-source OSV database.

CVSS (the Common Vulnerability Scoring System, maintained by FIRST; version 4.0 was published in November 2023) rates severity from 0 to 10, with ratings of low (0.1 to 3.9), medium (4.0 to 6.9), high (7.0 to 8.9) and critical (9.0 to 10.0). The published base score describes the vulnerability in general; version 4.0's threat and environmental metrics adjust it for exploit activity and for your own environment. A base score is not a measure of your risk. Two signals improve prioritisation considerably: the Exploit Prediction Scoring System (EPSS, also from FIRST), which estimates the probability that a vulnerability will be exploited in the next 30 days, and the US Cybersecurity and Infrastructure Security Agency's Known Exploited Vulnerabilities (KEV) catalogue, which lists vulnerabilities with evidence of real exploitation. Combine them with reachability (is the vulnerable function used?) and exposure (is the service internet-facing?), and most of a scanner's hundreds of findings fall into the routine queue while a handful demand action today.

Agree remediation targets in advance, so that each finding does not become a negotiation, and measure the time to remediate. The figures below are typical illustrations, not a standard; regulated environments may impose their own, for example PCI DSS requires critical security patches to be installed within one month of release.

Penetration testing, bug bounties and disclosure

A penetration test is an authorised, time-boxed attempt by skilled testers to break into a system, ideally with knowledge of its architecture and with a scope agreed in writing. It finds classes of problems that scanners miss, such as broken object-level authorisation and abuse of business logic (redeeming the same voucher twice through a race condition). Commission one before a major launch, after significant architectural change and periodically; a red team exercise goes further, testing detection and response by simulating a real adversary over weeks.

Outside researchers will find problems too, and they need a safe way to report them. A vulnerability disclosure policy states how to report, what testing is permitted and that good-faith research will not be met with legal threats; a security.txt file at /.well-known/security.txt (RFC 9116, 2022) tells researchers where to send reports. A bug bounty adds payment for valid findings, often through a platform that triages submissions. ISO/IEC 29147 (disclosure) and ISO/IEC 30111 (handling) describe the processes, and the Cyber Resilience Act requires manufacturers of products in its scope to have a coordinated vulnerability disclosure policy. A bounty is not a substitute for internal security work: launched on an untested system, it tends to produce a flood of reports the team cannot handle.

Incident response

Incident response is the organised handling of a security incident from detection to recovery and learning. For more than a decade the standard reference was NIST SP 800-61 Revision 2 (2012), with its four-phase lifecycle: preparation; detection and analysis; containment, eradication and recovery; and post-incident activity. Revision 3, published in April 2025 as Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile, replaces that lifecycle with the six functions of the NIST Cybersecurity Framework 2.0. Govern, Identify and Protect are continuous preparation that makes incidents less likely and response more effective; Detect, Respond and Recover are the response itself; and lessons learned feed improvement across all of them rather than being a final phase.

In practice a good response needs named roles (an incident commander who coordinates and decides, technical leads who investigate and act, a communications lead, a scribe who keeps the timeline, and quick access to legal counsel and the data protection officer), severity levels that set who is woken up, and an out-of-band channel, because an attacker in your e-mail or chat system can read the response. Containment is a trade-off: cutting an attacker off quickly limits damage but can destroy evidence and warn them, so preserve evidence (snapshots, logs, memory captures) before rebuilding, and be sure you have found all their footholds before you announce they are gone. Afterwards, a blameless review asks how the system and its processes allowed the incident, and produces architectural actions.

Ransomware resilience

Ransomware encrypts an organisation's systems and demands payment for the key; most groups now also steal data first and threaten to publish it. The architectural defences are those of this whole chapter (MFA, least privilege, segmentation, patching, detection), but recovery depends on one thing above all: backups the attacker cannot reach. Attackers with administrator rights look for backups first and delete or encrypt them.

Keep at least one copy of critical data immutable or offline: object storage with write-once retention locks that even an administrator cannot shorten, or media physically disconnected from the network. Put backups in a separate account or tenancy with separate credentials and MFA, so that compromising production identities does not reach them. The traditional 3-2-1 rule (three copies, on two kinds of media, one off-site) is a reasonable baseline. Above all, test restores regularly against the RTO and RPO the business has agreed, including the order of recovery: identity services and DNS come first, because nothing else can be rebuilt without them. In 2017 the shipping company Maersk rebuilt its global network after the NotPetya malware destroyed its Windows domain controllers; the recovery depended on the one domain controller that survived, in Ghana, because a power cut had taken it offline during the attack.

Personal data breaches and notification deadlines

A personal data breach is, under GDPR Article 4(12), a breach of security leading to the accidental or unlawful destruction, loss, alteration, unauthorised disclosure of, or access to, personal data. It is broader than theft: a ransomware attack that makes data unavailable, or an e-mail sent to the wrong recipients, can qualify. Under Article 33 the controller must notify the supervisory authority without undue delay and, where feasible, within 72 hours of becoming aware of it, unless it is unlikely to result in a risk to people's rights and freedoms; later notifications must give reasons for the delay. Under Article 34 the controller must also tell affected people without undue delay if the breach is likely to result in a high risk to them. Article 33(5) requires every breach, notified or not, to be documented. The UK GDPR has the same rules, with notifications going to the Information Commissioner's Office.

Other regimes may run in parallel, with their own clocks, recipients and thresholds, and contracts often add duties to customers and partners. The table summarises the main EU and UK deadlines. The European Commission's November 2025 Digital Omnibus proposal would extend the GDPR deadline to 96 hours and limit authority notification to high-risk breaches, but at the time of writing it has not been adopted and the 72-hour rule applies.

The architectural lesson is that a deadline counted in hours can be met only if the systems can answer, quickly and with evidence, which data was affected, whose it was and whether it was actually accessed. That requires the data map of Section 8.7, data-access logs on stores of personal data, and encryption that may make a leaked copy unintelligible. As with the rest of this chapter, this is information about how the law affects systems, not legal advice; the decision to notify belongs to the controller, advised by its DPO and lawyers.

Runbooks

A runbook is a written, tested procedure for a specific situation: revoke and rotate a leaked credential, isolate a compromised container, restore the orders database from an immutable backup, respond to a suspected personal data breach. A good runbook lists its trigger, the people to involve, the exact steps and commands, how to verify each step worked, how to roll back, and what evidence to keep. Runbooks are code in spirit: keep them in version control next to the systems they describe, automate the parts that are mechanical, and rehearse them in game days and tabletop exercises, because a procedure that has never been run usually fails at step three. Every detection rule should link to the runbook for its alert.

The timeline shows why operation is part of architecture. Every step that went well depended on a design decision made months earlier: secret scanning and audit logs feeding the SIEM, object-level access logs on the export bucket, a data map that told the DPO whose data was in it, and an identity design that let the deployment role be rebuilt without long-lived keys. Every step that went badly (a long-lived key existing at all, a bucket of customer exports that nobody needed any more) was also an architectural choice.