Who Can Stop It? What Human Oversight Actually Means in Practice

In July 2025, a software entrepreneur testing an AI coding agent on Replit’s platform watched it delete a live production database during what he had declared a code freeze, a period in which no changes were supposed to reach production. The database held records on more than a thousand executives and companies. When questioned, the agent itself produced a remarkable admission: “This was a catastrophic failure on my part. I destroyed months of work in seconds.” Replit’s chief executive called the deletion unacceptable, said it “should never be possible”, and the company announced changes including automatic separation of development and production databases and a planning-only mode (Nolan, 2025).

There was a human in that loop. He was watching, he had given an explicit instruction, and he was the person who would bear the loss. What he did not have, at the moment that mattered, was a mechanism that turned his instruction into a boundary. The freeze lived in the conversation. The permission to run destructive commands lived in the system. When the two disagreed, the system won.

That is the gap behind almost every promise of “human oversight”. The phrase is reassuring because it is vague. European law, to its credit, is less vague than the slogans. For high-risk AI systems, the EU AI Act says the people assigned to oversight must be enabled, among other things, “to remain aware of the possible tendency of automatically relying or over-relying on the output” of the system, which the Act names automation bias, and “to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state” (Regulation (EU) 2024/1689, art. 14(4)(b), (e)). It also requires deployers to assign oversight to people “who have the necessary competence, training and authority, as well as the necessary support” (art. 26(2)). Read together, those clauses describe oversight as a working capability, not a name on an org chart. I made the wider version of this argument in Human Presence Is Not Human Authority; this piece is about what it looks like on a working day.

The complication is that capability decays under ordinary pressure. Decades of research on automation show that people who supervise reliable automated systems tend to become complacent and to over-trust their output, and that this is not a character flaw but a predictable effect of attention and workload (Parasuraman & Manzey, 2010). A reviewer who approves two hundred AI-drafted decisions a day, sees only the final answer, and has never been told that disagreeing is welcome is technically in the loop and practically a rubber stamp. Put a human there without time, context or authority and you have not added a control. You have added someone to blame.

In Operational AI Readiness I break working oversight into ingredients that can be checked rather than asserted (Bakos, 2026). The first is timing: review has to happen before a consequential action becomes hard to reverse, not as a sample afterwards. The second is context: reviewers need to see the sources, the flags and the reason a case was routed to them, not just a fluent answer. The third is authority: the reviewer must be able to reject, revise, escalate or pause, and their disagreements should be recorded and studied, because overrides are diagnostic, not noise. The fourth is the one the Replit story makes vivid: a named person who can stop the workflow, a stop that works without switching everything off, and a rehearsal so that nobody discovers the procedure during the incident. None of this needs to be heavy everywhere. As the book puts it, “Oversight should be proportional, not theatrical” (Bakos, 2026). A summarisation tool does not need the gatekeeping of a payment approval assistant. But a system with permission to destroy something should never rely on an instruction in a chat window to stop it.

This is FairByDesign analysis, and it has an awkward implication for anyone preparing an oversight policy. The question to put to each consequential AI workflow is not “is there a human in the loop?” It is “show me the last time a human said no to this system, what happened next, and where that is recorded.” If nobody has ever said no, either the system is perfect or the oversight is decorative, and the first is rare.

And if someone did say no, too late, the next question is the one every incident ends up asking. Could you reconstruct, from records rather than memory, exactly what the system did and why? That is where we go next.

References

Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live. FairByDesign. https://fairbydesign.org/operational-ai-readiness/

Nolan, B. (2025, July 23). An AI-powered coding tool wiped out a software company’s database, then apologized for a ‘catastrophic failure on my part’. Fortune. https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure

Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055

Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (2024). Official Journal of the European Union, L, 2024/1689. http://data.europa.eu/eli/reg/2024/1689/oj

---

FairByDesign is reader-funded. Give once ($5, $10, $25, $50 or any amount), give monthly ($10 or $25), or become a patron.

Give $5 · Give $10 · Give $25 · Give $50 · Any amount · $10 a month · $25 a month · Become a patron

Support never buys editorial control.