Data governance, responsible AI and the EU AI Act

Responsible AI as an architectural concern

Responsible AI gathers the principles that most frameworks share: fairness, transparency, accountability, privacy, safety, security and robustness. Stated as principles they are hard to disagree with and easy to ignore. They become real only when they are turned into properties of the system: where data flows and how long it stays, what is logged and who can read it, where a person can intervene, what is documented and kept current, and what is measured for which groups. Those are architectural decisions, often expensive to retrofit, which is why responsible AI belongs in design reviews and decision records rather than in a policy document alone.

This section builds on the privacy principles of section 8.7 and the ethics discussion of section 11.8, and applies them to systems with learned, probabilistic components. It then turns to the main regulatory and standards frameworks an architect will meet: the EU AI Act, the NIST AI Risk Management Framework and ISO/IEC 42001.

Data governance: provenance, licences and personal data

Data provenance is the documented origin and history of data: where it came from, under what licence or consent, and how it was transformed. AI systems consume data in more ways than ordinary software: training and fine-tuning data shape the model's weights, retrieval corpora shape its answers, evaluation sets define what good means, and feedback loops can feed outputs back in as inputs. Each needs a known source and a lawful, licensed basis for its use. Model weights carry licences too: open-weight models are released under licences that range from permissive open-source licences to custom licences with acceptable-use restrictions or conditions for very large users, and those terms are constraints on the architecture just like a database licence.

Personal data deserves particular care because AI features spread it into new places. Every prompt sent to a hosted model is a disclosure to a processor, which under the GDPR needs a contract, a suitable location or transfer mechanism, and clarity about the provider's retention and whether it may use the data for training. Data minimisation applies to prompts: send the order number and the missing item, not the customer's full profile. Pseudonymise or redact where the task allows. The table shows the less obvious places personal data accumulates. The hardest is model weights: data used for fine-tuning can be memorised and cannot be selectively removed, so personal or fast-changing data belongs in a retrieval store (section 12.3), where it can be filtered, corrected and deleted, not in the weights.

Bias and fairness

Algorithmic bias means systematically worse outcomes for some groups of people. It enters through historical data that encodes past discrimination, through groups that are under-represented in the data, through proxies (a postcode standing in for ethnicity), through measurement choices (using arrests as a proxy for crime) and through deployment in contexts the model was not built for. Language models add their own forms: stereotyped associations learned from web text and markedly uneven quality across languages and dialects, so that a support assistant may be excellent in English and poor in Polish or Welsh.

Fairness can be defined in several measurable ways, for example equal rates of positive decisions across groups (demographic parity), equal error rates across groups (equalised odds) or scores that mean the same probability in every group (calibration). Kleinberg, Mullainathan and Raghavan (2016) and Chouldechova (2017) showed that when groups have different base rates, a classifier generally cannot satisfy calibration and equal error rates at the same time. Choosing a fairness criterion is therefore a value judgement for the people affected and the organisation, not a technical default. The architect's job is to make the choice explicit, to slice evaluation results by group and language as in section 12.5, and to decide how the data needed for that measurement is collected and protected. Measuring bias often requires sensitive attributes that the system otherwise should not hold; Article 10(5) of the AI Act allows their processing for bias detection in high-risk systems under strict safeguards, and the 2026 amendments added a similar, strictly conditioned permission for other AI systems and models.

Transparency, explainability and model cards

Transparency is telling people what they are dealing with: that they are talking to an AI, that an image is synthetic, what a system is for and where it is unreliable. Explainability is being able to say why a particular output was produced. For classical machine-learning models, techniques such as feature attributions can indicate which inputs influenced a score, with known limitations. For language models, explanations are harder: a model's own account of its reasoning is generated text that need not reflect the computation that actually produced the answer. Practical explainability in LLM systems therefore comes from architecture rather than introspection: show the retrieved sources an answer relied on, keep consequential decisions in deterministic rules that produce reason codes, and log the inputs, sources, prompt and model versions so that any decision can be reconstructed and contested later. Where decisions with legal or similarly significant effects are made solely by automated means, the GDPR adds rights to human intervention and to meaningful information about the logic involved.

Model cards, proposed by Mitchell and colleagues in 2019, are short documents that accompany a trained model: its intended uses and out-of-scope uses, the factors (such as demographic groups) across which it was evaluated, its metrics and their results per group, its training and evaluation data, ethical considerations and caveats. Datasheets for datasets (Gebru et al.) do the same for data, and many providers now publish system cards covering a whole deployed system. For architects these are both inputs (read the card of every model you adopt, including its limits and licence) and outputs (publish one for your own system, kept in the repository and updated with each release, alongside the architecture documentation of Chapter 4).

Human oversight

Human oversight takes three common forms: a person approves each decision before it takes effect (human in the loop), a person monitors the system and can intervene (human on the loop), or people govern when and how the system is used at all (human in command). Oversight is only as good as its design. Automation bias, the tendency to accept a system's recommendation without real scrutiny, turns reviewers who approve hundreds of AI suggestions an hour into a rubber stamp that adds liability but no safety.

Effective oversight is designed in: show the evidence and sources behind each recommendation, not just the answer; make disagreement easy and record it; size review workloads realistically; periodically insert known-wrong cases to check that reviewers catch them; measure override rates (a rate near zero is a warning sign, not a success); and give operators a tested way to pause or stop the system. Section 12.2 discusses human-in-the-loop designs and their capacity planning. Article 14 of the AI Act requires high-risk systems to be designed so that overseers can understand the system's capacities and limits, remain aware of automation bias, interpret outputs, decide not to use or to override them, and interrupt the system.

The EU AI Act

The EU AI Act, Regulation (EU) 2024/1689, is the first comprehensive horizontal law on AI. It entered into force on 1 August 2024 and applies in stages. It regulates by risk, assigns duties mainly to providers (who develop a system or model and place it on the market under their name) and deployers (who use it professionally), and reaches beyond the EU: it applies to providers placing systems on the EU market and to providers and deployers elsewhere whose systems' output is used in the EU. Penalties for prohibited practices reach EUR 35 million or 7% of worldwide annual turnover, whichever is higher, with lower ceilings for other breaches.

The Act sorts systems into four tiers, plus a separate track for models. Prohibited practices (Article 5) include social scoring, manipulative or deceptive techniques and exploitation of vulnerabilities that cause significant harm, untargeted scraping of facial images, emotion recognition in workplaces and schools (with narrow exceptions), biometric categorisation to infer sensitive characteristics, predicting criminal offences solely from profiling, and real-time remote biometric identification in public spaces for law enforcement except in narrow cases. High-risk systems (Article 6) are safety components of products covered by the EU product-safety legislation listed in Annex I, and systems used in the areas listed in Annex III: biometrics, critical infrastructure, education, employment and worker management, access to essential private and public services (such as credit scoring and life and health insurance pricing), law enforcement, migration and border control, and justice and democratic processes. An Annex III system can escape the high-risk category under Article 6(3) if it performs only a narrow or preparatory task and poses no significant risk, but never if it profiles people. Transparency obligations (Article 50) require people to be told when they interact with an AI system unless that is obvious, generated or manipulated content to be marked in a machine-readable way, and deepfakes to be disclosed. Minimal-risk systems, the large majority, have no specific new obligations, although the AI literacy provision of Article 4 applies to providers and deployers generally.

General-purpose AI (GPAI) models, such as the large language models behind most assistants, are regulated at the model level. Under Article 53 their providers must keep technical documentation, give downstream providers the information they need, have a policy to comply with EU copyright law and publish a summary of the content used for training. Models with systemic risk, presumed when training compute exceeds 10²⁵ floating-point operations, carry further duties under Article 55: evaluation including adversarial testing, risk mitigation, serious-incident reporting and cybersecurity. A General-Purpose AI Code of Practice, published in July 2025, offers providers a voluntary way to demonstrate compliance. A company that builds an application on someone else's model is normally a provider or deployer of an AI system, not of the model, but may become a model provider if it substantially modifies the model.

The timeline and the 2026 amendments

The Act's obligations were scheduled to arrive in stages between February 2025 and August 2027. In November 2025 the European Commission proposed a Digital Omnibus on AI to simplify implementation, chiefly because the harmonised standards and guidance needed for the high-risk rules were late. After agreement between the European Parliament and the Council, it was adopted as Regulation (EU) 2026/1744 of 8 July 2026, published in the Official Journal on 24 July 2026 and in force since 27 July 2026. It postpones the high-risk obligations to fixed dates (2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems); adds prohibitions on AI systems that generate or manipulate child sexual abuse material or intimate images of identifiable people without their consent; gives providers of generative systems already on the market a four-month transitional period for machine-readable marking; changes the AI literacy duty from ensuring a sufficient level of literacy to taking measures to support it; adds a legal basis, under strict conditions, for processing special categories of personal data to detect and correct bias beyond high-risk systems; extends simplifications from SMEs to small mid-cap enterprises; moves the Machinery Regulation from Section A to Section B of Annex I, so that requirements for AI in machinery are to be applied through that regulation rather than directly under the Act; and gives the AI Office a stronger role in supervising AI systems built on general-purpose models. The prohibitions and GPAI obligations already in force were not postponed, and the Article 50 transparency obligations apply from 2 August 2026 as planned.

A separate Digital Omnibus proposal from November 2025, which would amend the GDPR, the ePrivacy rules and the Data Act, follows its own legislative process; check whether it has been adopted, and design against the law in force, not against proposals. Check the current consolidated text of the AI Act before relying on any date in the table below.

What does this mean for the food-delivery platform? The customer-support assistant is not in an Annex III area, so it is not high-risk, but it must tell customers they are talking to an AI, and the GDPR applies in full to its prompts and logs. The same company's courier systems are a different matter: AI that allocates work to couriers on the basis of their behaviour or characteristics, monitors and evaluates their performance or decides on deactivation falls within Annex III's employment and worker-management area, and therefore within the high-risk regime from December 2027, as well as the EU Platform Work Directive (Directive (EU) 2024/2831) on algorithmic management, which member states must transpose by December 2026. Classification follows the purpose, not the technology: one model can serve a minimal-risk use and a high-risk use in the same company.

Frameworks and standards: NIST AI RMF and ISO/IEC 42001

The NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1, January 2023) is a voluntary US framework, widely used internationally. It organises AI risk management into four functions: Govern (culture, policies, roles and accountability), Map (context, intended use and potential impacts), Measure (analysis, testing and tracking of risks) and Manage (prioritising and treating risks, and responding to incidents). It describes trustworthy AI as valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. Its Generative AI Profile (NIST AI 600-1, July 2024) adds risks and actions specific to generative models. Check NIST's site for the current version and its companion profiles before citing it.

ISO/IEC 42001:2023 specifies requirements for an AI management system: the policies, roles, risk and impact assessments, controls, monitoring and continual improvement through which an organisation governs its AI work, in the same Plan-Do-Check-Act style as ISO/IEC 27001 for information security. Unlike the NIST framework it is certifiable, and organisations increasingly ask suppliers for certification. ISO/IEC 23894:2023 gives guidance on AI risk management. The three fit together: the AI Act states legal obligations, harmonised European standards (under development by CEN and CENELEC) will describe how to meet them, and the NIST framework and ISO/IEC 42001 provide the management practices in which the evidence is produced.