The Vanishing Human
It is late on a Friday. The system has flagged the case, drafted the outcome and put a button in front of you. Your name goes on the approval. If the machine was wrong, the record will say a human decided. (That Friday is an illustration. The pattern behind it is not.)
Researchers have a name for where you are sitting. Madeleine Clare Elish showed that, when automated systems fail, blame tends to fall on the nearest human, who becomes a “moral crumple zone” for the system (Elish, 2019). The institution gets the speed. You absorb the impact.
We have been hard on the phrase “human in the loop.” In our piece on human presence and human authority, we argued that a person who cannot see, understand, refuse or stop a system is a witness, not an overseer. Now I want to put that argument in the dock and call its two strongest opponents. They come from opposite directions. One says we are not radical enough: individual human oversight of AI is a fiction and should be abandoned. The other says we are too purist: the fiction is useful and worth keeping, even when it is imperfect.
If you are the person whose name goes on the approval, both arguments are about you.
Witness one: scrap the fiction
The first argument goes further than we did, and it starts with the evidence about people. Lisanne Bainbridge showed in 1983 that automation can leave humans responsible for monitoring and for dealing with abnormal conditions, while making those responsibilities harder to perform when they are suddenly required to take over (Bainbridge, 1983). Parasuraman and Manzey’s review found automation complacency and bias in experts as well as novices, and found them hard to train away (Parasuraman & Manzey, 2010). In other words, the person in the chair is set up to miss things, and then set up to take the blame.
Ben Green draws the hard conclusion. Having reviewed 41 policies requiring human oversight of government algorithms, he argues that such policies “provide a false sense of security,” because people cannot do what the policies expect of them, and the policies therefore legitimise systems that should not have been adopted (Green, 2022). His proposed alternative is not better individual oversight. It is institutional oversight: before adoption, agencies would have to justify both the algorithm and any human role in it with evidence, and submit that justification to democratic review.
Follow that logic and it lands a punch on us. Demanding “meaningful” human control, as we did, may be a trap. It keeps the institution’s attention on the person in the chair when it should be on the decision to deploy. A better stop button does not help if the system should never have been switched on. On this view, our five questions about your own loop are the wrong questions. The right one is whether the system should exist in this setting at all, and who had the power to answer that.
Witness two: keep the fiction, it has teeth
The second argument runs the other way. Legal fictions are everywhere, and many are useful. A corporation is not a person, yet treating it as one lets us sue it. On this view, the requirement that a human oversee a high-risk AI system is valuable even if real individuals often fall short, because it creates a legal hook. The EU AI Act’s Article 14 does not just say a human must be present. It requires systems to be designed so they “can be effectively overseen by natural persons,” with the ability to override and a way to halt the system (Regulation (EU) 2024/1689, 2024, Art. 14). The requirement forces providers to build interfaces, stop mechanisms and documentation they might otherwise skip. Even an imperfect overseer creates a record, a point of contact and someone with standing to raise the alarm.
Scholars who defend oversight make a careful version of this case. Sterz and colleagues argue that effective oversight is achievable when the overseer has causal power over the system, epistemic access to it, self-control and fitting intentions. On that view the task is to engineer those conditions, not to give up (Sterz et al., 2024). Laux argues for oversight as institutionalized distrust: a system of checks and balances designed around the fallibility of individual overseers, rather than trust in one heroic individual (Laux, 2024). Enqvist’s legal analysis of the Commission’s proposal for the Act points to unclarities and gaps in its oversight requirement, examining what is to be overseen, when oversight is exercised and by whom, and warns against vesting too much trust in providers to build the oversight infrastructure (Enqvist, 2023). The defender’s response is that you fix a vague obligation by sharpening it, not by deleting it.
And this witness also has a punch for us. By insisting that oversight without full authority is “only presence,” we risk telling deployers that partial oversight is worthless, which gives them a reason to drop the human entirely. Sometimes an imperfect check that occasionally catches something beats none at all. That one stings, because it might be true.
My verdict, for now
This is FairByDesign’s analysis. Both witnesses are partly right, and they are less opposed than they look. Green is right that the most important oversight decision happens before deployment, and that an individual approving outputs cannot carry the weight of a bad adoption decision. The defenders are right that the legal requirement for a human who can override and halt a system is a lever worth keeping, because it forces design choices and creates accountability.
Where I still disagree with the “keep the fiction” side is on the word fiction. A requirement everyone knows to be fictional teaches organisations that compliance is a performance, and it leaves you, the person in the chair, holding the blame for a decision you could not really make. The useful version of the requirement is the one that is tested. Can this person, in this workflow, with this time and information, actually stop this system? If nobody checks, the lever exists only on paper.
What we still don’t know
We are holding that view provisionally. We don’t know how often, in real workplaces, the human in the loop actually stops a system, and none of the research above answers that for your workplace. We don’t know whether a tested oversight requirement would make deployers more careful or simply push them to drop the human altogether. Those are the answers that would change our minds, and we would like to be argued out of our view if we are wrong.
You’ve been the human in the loop. Tell us where we’re wrong
This is an open invitation, especially to people who do this work: operators, reviewers, caseworkers, clinicians, moderators, engineers, lawyers and researchers who have been the human in the loop, or designed one. Is human oversight of AI a fiction worth keeping, a fiction worth abandoning, or something real we have described badly? Send your response, your evidence or your story to info@fairbydesign.org, or send sensitive material end-to-end encrypted to viktoriabakos@proton.me. With your permission, we will publish the strongest responses, including those that disagree with us, and discuss them on the podcast.
For the practitioner’s treatment of oversight design, see Chapter 5 of our free field manual, Operational AI Readiness.
Why this needs independent support: someone has to ask whether the human in the loop can really say no, and be willing to publish the answer when it complicates their own argument. Give once ($5, $10, $25, $50 or any amount), give monthly ($10 or $25), or become a patron → fairbydesign.org/patrons-2/
References
Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779. https://doi.org/10.1016/0005-1098(83)90046-8
Elish, M. C. (2019). Moral crumple zones: Cautionary tales in human-robot interaction. Engaging Science, Technology, and Society, 5, 40–60. https://doi.org/10.17351/ests2019.260
Enqvist, L. (2023). ‘Human oversight’ in the EU artificial intelligence act: What, when and by whom? Law, Innovation and Technology, 15(2), 508–535. https://doi.org/10.1080/17579961.2023.2245683
Green, B. (2022). The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review, 45, Article 105681. https://doi.org/10.1016/j.clsr.2022.105681
Laux, J. (2024). Institutionalised distrust and human oversight of artificial intelligence: Towards a democratic design of AI governance under the European Union AI Act. AI & Society, 39(6), 2853–2866. https://doi.org/10.1007/s00146-023-01777-z
Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (2024). Official Journal of the European Union, L, 2024/1689. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
Sterz, S., Baum, K., Biewer, S., Hermanns, H., Lauber-Rönsberg, A., Meinel, P., & Langer, M. (2024). On the quest for effectiveness in human oversight: Interdisciplinary perspectives. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (pp. 2495–2507). ACM. https://doi.org/10.1145/3630106.3659051




Leave a Reply
You must be logged in to post a comment.