What Is AI Governance? A Plain-Language Guide

In the summer of 2020, England cancelled its school exams because of the pandemic. Teachers were asked to estimate the grade each student would most likely have achieved and to rank their students, and those estimates were then “standardised” using a statistical model developed with Ofqual, the exams regulator (Department for Education, 2020b). On 13 August, A-level students opened their results. According to Ofqual figures reported by The Next Web, the model had lowered about 39 percent of the grades teachers had estimated, in part by weighing each school’s past results (Macaulay, 2020).

Four days later, on 17 August, the government and Ofqual abandoned the calculated grades. Students would receive their teachers’ grades instead, unless the model had given them something higher. The official announcement said the process had produced “too many inconsistent and unfair outcomes” and that the algorithm had revealed “a number of anomalies that had not been anticipated by Ofqual” (Department for Education, 2020a).

This is not a story about a bad computer. It is a story about decisions made around one: what the model was for, who checked how it would treat different kinds of students, what a student could do if the result was wrong, and who had the authority to stop it. Those questions have a name. They are AI governance.

What the phrase actually means

“AI governance” sounds like something that happens in boardrooms. In practice it is much more ordinary. It is how an organization decides which AI systems it uses, for what, within which limits, how it keeps watch on them once they are running, and who answers when something goes wrong.

A useful way to hold it in your head is three Rs.

Rules are the limits a system must work within. Some come from law, such as the European Union’s AI Act. Some come from voluntary frameworks, such as the US National Institute of Standards and Technology’s AI Risk Management Framework, which NIST describes as intended for voluntary use (NIST, 2023, p. 2). Some come from standards like ISO/IEC 42001, which sets out how an organization can run an “AI management system” of policies, objectives and processes (ISO, 2023). And some are the organization’s own: this tool may draft letters, but it may not send them.

Roles are the named people. Someone owns each system. Someone reviews the outputs that matter. Someone can say no.

Routines are what happens every week: testing before launch, watching results after launch, recording what the system did and what changed, and a way to pause it when the results look wrong.

Governance is a practice, not a document

Most organizations that use AI now have something written down: principles, an acceptable-use policy, an ethics statement. Those are a start. But a document cannot tell you what a system did last Tuesday, and it cannot stop one this afternoon.

NIST makes the same point in its own way. Its framework treats “Govern” not as a separate box to tick but as a function that runs through everything else, from understanding a system’s context to measuring and managing its risks (NIST, 2023, pp. 20–21). Our free field manual, Operational AI Readiness, puts it more bluntly: for systems that influence consequential decisions, “governance is not a policy document sitting politely upstream from engineering” (Bakos, 2026). A system with no monitoring, no change control and no way to stop it may still work, “but it is not operationally ready. It is just active” (Bakos, 2026).

Go back to 2020. The difficult questions were not mainly about code. Had anyone looked closely at how the model would treat a strong student at a school with weak past results? What could that student do on results day, and how quickly? Who was watching the pattern of outcomes, and who could pull the plug? The answers arrived in public, under pressure, after the results were out. Good governance tries to answer them before.

Why it matters to you

You will rarely see the AI that affects you. It may sort job applications, flag a payment, prioritise a call, draft a doctor’s note or suggest who should be checked for fraud. When it works, nobody notices. When it goes wrong, the person affected is often the last to know, and the first question they need answered is a governance question: was a machine involved, and can a person look at this again?

Governance is also what turns a complaint into a correction. In 2020, students and families could see their results and push back, and the decision was reversed within days. Many automated decisions are far less visible than an exam grade, which is why rules about records, review and the right to challenge matter so much.

Why it matters to organizations

For organizations, the honest answer is that most of the risk is not in the model itself but in how it is used: the data it is fed, the people who rely on it, the changes nobody logged, the vendor nobody questioned. That is the territory Chapter 2 of Operational AI Readiness maps.

The rules are also arriving. The EU AI Act entered into force in August 2024, its bans on certain practices have applied since February 2025, and, after an amendment that took effect in July 2026, its main rules for stand-alone high-risk systems apply from 2 December 2027 (European Commission, n.d.). Even where no law applies, customers, insurers, auditors and courts increasingly ask the same thing: show us how you know this system is doing what you say it does.

What the record does not show

The public record of 2020 shows the outcome and the reversal. It does not show, in the sources we opened, exactly which checks were done before results day or what a different process would have caught. The headline percentage counts grades, not students. The model was a statistical standardisation method rather than the kind of machine-learning system many people now mean by “AI”, and it was designed for a real purpose: keeping grades comparable across schools in an extraordinary year. We use it here because the governance questions are the same, whatever the technology.

Questions you can ask

• If a decision affects you: Was an automated system involved? Can a person review the decision, and how do I ask?
• If a decision affects you: What information about me was used, and how do I correct it if it is wrong?
• If you work somewhere that uses AI: Do we have a list of the AI systems we use, and does each one have a named owner?
• If you work somewhere that uses AI: What result would make us pause this system, and who is allowed to do it?

This is the first post in AI Governance 101, a plain-language series for anyone new to these questions. Next week: when an AI system helps decide, who is actually responsible? Get the free Mercury Brief and we will send each new explainer to your inbox.

---

FairByDesign is reader-funded. If this helped, get the free Mercury Brief or see ways to support. Support never buys editorial control.

References

Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live. FairByDesign. https://fairbydesign.org/operational-ai-readiness/

Department for Education. (2020a, August 17). GCSE and A level students to receive centre assessment grades [News story]. GOV.UK. https://www.gov.uk/government/news/gcse-and-a-level-students-to-receive-centre-assessment-grades

Department for Education. (2020b). Coronavirus (COVID-19): Cancellation of GCSEs, AS and A levels in 2020 [Withdrawn guidance, last updated August 27, 2020]. GOV.UK. https://www.gov.uk/government/publications/coronavirus-covid-19-cancellation-of-gcses-as-and-a-levels-in-2020/coronavirus-covid-19-cancellation-of-gcses-as-and-a-levels-in-2020

European Commission. (n.d.). AI Act [Regulatory framework for AI]. Shaping Europe’s digital future. Retrieved October 15, 2026, from https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai

International Organization for Standardization. (2023). ISO/IEC 42001:2023 Information technology — Artificial intelligence — Management system. https://www.iso.org/standard/42001

Macaulay, T. (2020, August 13). Algorithm downgrades nearly 40% of English exam results, hitting poorer students hardest. The Next Web. https://thenextweb.com/news/algorithm-downgrades-nearly-40-of-english-exam-results-hitting-poorer-students-hardest

National Institute of Standards and Technology. (2023). Artificial intelligence risk management framework (AI RMF 1.0) (NIST AI 100-1). U.S. Department of Commerce. https://doi.org/10.6028/NIST.AI.100-1