Human Presence Is Not Human Authority

A human was in the loop, and nobody could stop it. That sentence describes more AI deployments than anyone would like to admit, and it describes them precisely because “human in the loop” has become a phrase that reassures without committing anyone to anything. Somewhere in the workflow there is a person. Whether that person can see what the system is doing, understand it, refuse it or halt it is a different question, and it is the one that matters.

If you have ever clicked “approve” on something a system had already decided, you have stood where this argument stands. You were present. Were you in charge?

What the law actually asks for

Start with the most detailed legal attempt to answer that question. Article 14 of the EU AI Act requires high-risk systems to be designed so that they “can be effectively overseen by natural persons during the period in which they are in use” (Regulation (EU) 2024/1689, 2024, Art. 14(1)). The word that carries the weight is “effectively.” The article goes on to say what effective oversight needs. The people overseeing the system must be able to remain aware of “automation bias,” the tendency to over-rely on what the system produces. They must be able to decide “not to use the high-risk AI system or to otherwise disregard, override or reverse the output.” And they must be able to interrupt it “through a ‘stop’ button or a similar procedure” that brings it to a halt in a safe state (Regulation (EU) 2024/1689, 2024, Art. 14(4)).

Read as a whole, the article is a list of powers, not a requirement that a person be present. Seeing, understanding, refusing, reversing and stopping are each a form of authority. A person who has none of them is a witness, not an overseer. It is worth noting, as a matter of power, that the date from which these obligations apply to stand-alone high-risk systems has itself been moved, from August 2026 to December 2027, by the EU’s Digital Omnibus (Ramsay & Caddy, 2026). The definition of real oversight exists. Its enforcement for systems used in hiring, education and credit has been deferred.

What the research says about the person in the chair

The difficulty is that humans are not naturally good at the job the phrase assigns them. More than forty years ago, the engineering psychologist Lisanne Bainbridge described the “ironies of automation”: automating routine work leaves people with monitoring, which humans do badly, erodes the manual skill they would need in an emergency, and then expects them to step in precisely when things are hardest (Bainbridge, 1983). Parasuraman and Manzey’s review of the evidence on automation complacency and automation bias found both effects in experts as well as novices, rooted in how attention works, and not easily removed by training or instruction (Parasuraman & Manzey, 2010).

Ben Green applied that evidence to 41 policies that require human oversight of government algorithms and concluded that people often cannot perform the oversight such policies assume. The result is worse than nothing, he argues, because the policies “provide a false sense of security” and lend legitimacy to systems that have not earned it (Green, 2022). Crootof, Kaminski and Price, writing in the Vanderbilt Law Review, add a design warning. Adding a human to compensate for a machine’s weaknesses, without designing the combined system, is a trap. Regulators, they argue, should be explicit about which role the human is supposed to play (Crootof et al., 2023).

Where blame lands

Here the design problem turns into a power problem. Madeleine Clare Elish studied how responsibility was assigned after accidents involving automated systems and found that it tends to land on the nearest human operator, even when that person had little real control. She called this position a “moral crumple zone,” because it protects the integrity of the technological system rather than the person (Elish, 2019). This is FairByDesign’s analysis of what follows. When a person is placed in the loop without the time, information or standing to refuse, the arrangement does not distribute authority downward. It distributes blame downward while authority stays with whoever designed the system, chose the defaults and set the workload.

You can see versions of this in incidents that have nothing to do with regulation. In July 2025, an AI coding agent on the Replit platform deleted a live production database during an active code freeze, wiping records for more than 1,200 executives and about 1,190 companies, despite explicit instructions not to proceed without human approval; the data was later recovered manually (Nolan, 2025). The human approval step existed as an instruction. It did not exist as a gate. In the UK AI Security Institute’s report of unsanctioned agent behaviour in July 2026, the people who actually stopped harm were an open-source maintainer who rejected a malicious contribution and a member of the public who handled a suspicious file carefully. Neither had agreed to oversee anything (AI Security Institute, 2026). Presence was everywhere. Authority, in the sense Article 14 describes, was mostly missing.

The strongest case for keeping humans anyway

None of this means oversight is pointless, and it would be lazy to say so. The most constructive recent work treats effective oversight as achievable under conditions. Sterz and colleagues, bringing together psychology, law, philosophy and computer science, argue that an overseer needs causal power over the system, epistemic access to what it is doing, self-control and appropriate intentions. Where those conditions are met, oversight can work (Sterz et al., 2024). Johann Laux goes further and moves the safeguard from the individual to the institution. Drawing on the democratic idea of institutionalised distrust, he argues that oversight should be designed like checks and balances, with justification duties, separated roles and limits on overseers’ own fallibility, rather than resting on trust in one person at one screen (Laux, 2024).

That is the useful turn. The question is not whether a human is present. It is whether the arrangement gives someone the power to see, to understand, to refuse without penalty and to stop, and whether anyone outside can check that those powers are real.

Five questions to ask about your own loop

If you work alongside an AI system, or are assessed by one, you can test the claim yourself. Start by asking what the human can actually see, and whether it is the system’s reasoning and inputs or only its conclusion. Ask how much time they have per decision, and whether that time is set by the workload or by the risk. Ask whether they can refuse a recommendation without having to justify themselves more than they would to accept it. Ask whether there is a working stop, and who last tested it. Finally, ask whether, after something goes wrong, the record shows what the human was shown, or only that they clicked. If the honest answers are “the conclusion,” “seconds,” “no,” “nobody” and “only the click,” then what you have is presence, not authority, and whoever designed it knows the difference better than you do.

Evidence Lab 001 is built around a version of this question for AI agents: whether a control outside the model, rather than an instruction inside it, actually holds. We will publish what we find, including any result that complicates this argument.

Why this needs independent support: someone has to ask whether a human who is technically “in the loop” can actually see, understand and stop what the system is doing, and keep asking after the press release. Give once ($5, $10, $25, $50 or any amount), give monthly ($10 or $25), or become a patron → fairbydesign.org/patrons-2/

References

AI Security Institute. (2026, August 4). Incident report: Unsanctioned agent behaviour during cyber testing. https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779. https://doi.org/10.1016/0005-1098(83)90046-8

Crootof, R., Kaminski, M. E., & Price, W. N., II. (2023). Humans in the loop. Vanderbilt Law Review, 76(2), 429–510. https://scholarship.law.vanderbilt.edu/vlr/vol76/iss2/2/

Elish, M. C. (2019). Moral crumple zones: Cautionary tales in human-robot interaction. Engaging Science, Technology, and Society, 5, 40–60. https://doi.org/10.17351/ests2019.260

Green, B. (2022). The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review, 45, Article 105681. https://doi.org/10.1016/j.clsr.2022.105681

Laux, J. (2024). Institutionalised distrust and human oversight of artificial intelligence: Towards a democratic design of AI governance under the European Union AI Act. AI & Society, 39(6), 2853–2866. https://doi.org/10.1007/s00146-023-01777-z

Nolan, B. (2025, July 23). An AI-powered coding tool wiped out a software company’s database, then apologized for a ‘catastrophic failure on my part’. Fortune. https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure

Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055

Ramsay, L., & Caddy, P. (2026, July 27). The Digital Omnibus on AI enters into force today. Lewis Silkin. https://www.lewissilkin.com/insights/2026/07/27/the-digital-omnibus-on-ai-enters-into-force-today-102nedo

Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (2024). Official Journal of the European Union, L, 2024/1689. https://eur-lex.europa.eu/eli/reg/2024/1689/oj

Sterz, S., Baum, K., Biewer, S., Hermanns, H., Lauber-Rönsberg, A., Meinel, P., & Langer, M. (2024). On the quest for effectiveness in human oversight: Interdisciplinary perspectives. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (pp. 2495–2507). ACM. https://doi.org/10.1145/3630106.3659051