Every organisation now uses AI somewhere, often in more places than its leaders realise. AI governance is how an organisation makes sure that every one of those systems has a clear purpose, a named owner, appropriate safeguards and evidence that it works as intended. It is the difference between AI that people can trust and AI that becomes tomorrow's headline.
By the end of this chapter you can
- Define AI governance and distinguish it from ethics, compliance and risk management.
- Describe the five layers of a governance system and the roles in an AI operating model.
- Explain the three lines model, an AI inventory and risk-based tiering.
- Place governance gates across the AI lifecycle, including for AI bought from vendors.
What AI governance is, and what it is not
AI governance is the system of principles, policies, roles, processes and controls that directs how an organisation designs, buys, uses and retires AI, so that it is lawful, safe, fair and aligned with the organisation's purpose. It answers four practical questions for every AI system: Should we do this? Who is accountable? What could go wrong? How do we know it is working?
People often mix up four related ideas. They work together, but they are not the same:
AI ethics
What we believe is right: fairness, dignity, autonomy. Ethics sets direction but does not, on its own, say who must do what.
AI governance
How decisions about AI are made and enforced: structures, accountability, policies, oversight and evidence.
Compliance
Meeting the requirements of laws, regulations, contracts and standards. Necessary, but the minimum, not the goal.
Risk management
Identifying, measuring and treating what could go wrong. The engine inside governance (Chapter 6).
Why it matters now
AI changes the governance challenge in four ways. It operates at scale (one flawed model can affect millions of decisions), it is often opaque (even its builders cannot fully explain individual outputs), it changes over time (drift, retraining, new data), and it is increasingly regulated (Chapter 7). Two well-known cases show what happens when governance is missing:
Lessons from real failures
When nobody owned the risk
- Netherlands childcare benefits (2010s): tax authorities used risk-scoring that wrongly flagged thousands of families as fraudsters, with nationality among the risk factors used. Families were pushed into debt, and the scandal led the Dutch government to resign in January 2021. The failures were governance failures: no meaningful human oversight, no route to challenge decisions and no one checking for discrimination.
- An AI recruiting tool (2018): a large technology company scrapped an experimental CV-screening model after discovering it penalised CVs that mentioned women's activities, because it had learned from a decade of male-dominated hiring data. Testing caught it, which is governance working, but only after significant investment.
The five layers of an AI governance system
Mature organisations build governance in layers, from high-level intent down to evidence that can be audited. Each layer makes the one above it real.
Swipe sideways to see the full diagram.
The AI operating model: who does what
Governance fails when "everyone" is responsible, because then no one is. A clear operating model names the roles:
| Role | Main responsibility |
|---|---|
| Board / leadership | Sets the AI strategy and risk appetite (how much risk is acceptable), and receives regular reports. |
| AI governance council | A cross-functional group (technology, legal, risk, data protection, business, ethics) that sets policy and approves high-risk use cases. |
| AI system owner | A named business leader accountable for one system's purpose, performance and harms throughout its life. |
| Builders | Data scientists and engineers who design, test and document the system to the required standards. |
| Risk, compliance & data protection | Independent review, impact assessments and legal checks before and after launch. |
| Internal audit | Periodically checks, independently, that the whole system actually works as described. |
Many organisations organise these roles using the three lines model: the first line (teams building and running AI) owns and manages the risk; the second line (risk, compliance, data protection) sets standards and challenges the first line; the third line (internal audit) provides independent assurance to the board.
Centralised model
- One central AI team approves everything.
- Consistent and easy to control.
- Can become a bottleneck as AI use grows.
Hub-and-spoke model
- A central hub sets policy, tools and training; business units ("spokes") run their own AI within those rules.
- Scales better; the most common choice in large organisations.
Know what you have: the AI inventory
You cannot govern what you cannot see. The first practical step is an AI inventory (or register): a living list of every AI system the organisation builds, buys or uses, including AI features hidden inside software it already pays for. For each system, record at least:
Purpose & owner
What decision or task it supports, and the accountable person.
Data & model
Data sources (including personal data), model type, vendor and version.
Risk tier & status
Low, medium or high risk; in development, live or retired; last review date.
Risk tiering keeps governance proportionate. A tool that suggests email subject lines needs light checks; a system that influences credit, hiring, healthcare or public benefits needs a full impact assessment, independent testing, human oversight and senior approval. Spending the same effort on everything either slows innovation or leaves the dangerous systems under-protected.
Governance gates across the AI lifecycle
Governance is most effective, and cheapest, when it is built into the lifecycle as a series of gates: decision points where a system must show evidence before moving forward.
Swipe sideways to see the full diagram.
Governing AI you buy, not just AI you build
Much of the AI an organisation uses comes from vendors: chatbots, HR screening tools, fraud detection, AI features inside office software. Responsibility does not transfer with the invoice. Procurement governance asks vendors the hard questions before signing:
- What data was the model trained on, and does it include our data or our customers' personal data?
- What testing for accuracy, bias and security has been done, and can we see the results?
- Can we explain individual outputs to affected people, and audit the system?
- Where is data processed and stored, and how long is it kept?
- What happens when the model is updated, and will we be told in advance?
- Who is liable when the system causes harm, and how are incidents reported?
People: AI literacy is a governance control
Policies only work if people understand them. Staff who use AI need to know its limits (hallucination, bias), what data they must never paste into public tools, and when to escalate. The EU AI Act even makes AI literacy a legal obligation for organisations that provide or use AI systems. Training is not a soft extra; it is one of the most effective controls you have.
A starter AI policy in ten linesGo deeper
- Every AI system has a named, accountable owner.
- All AI systems are recorded in the AI inventory before use.
- Each system is risk-tiered; controls are proportionate to the tier.
- High-risk systems require an impact assessment and council approval.
- Personal data is used only with a lawful basis and minimised.
- Systems are tested for accuracy, fairness, robustness and security before launch.
- People are told when they interact with AI or are significantly affected by it.
- High-impact decisions keep meaningful human oversight and a way to appeal.
- Live systems are monitored; incidents are reported and learned from.
- Staff receive AI literacy training appropriate to their role.
Key takeaways
- AI governance turns values into decisions: who decides, who is accountable and what evidence is required.
- Build it in five layers: principles, policies, roles, processes and tools/evidence.
- Name owners, use the three lines model, keep an AI inventory and tier systems by risk.
- Place gates across the lifecycle, govern purchased AI as seriously as built AI, and treat AI literacy as a control.