Episode 4: The Human in the Loop Who Can’t Say No

“There’s a human in the loop” is meant to end the conversation. In episode 4, Viktoria Bakos uses the NTSB’s findings on the 2018 Uber test-car crash to show why a human who sees too little, too late, with no…

FairByDesign Operational AI Readiness title card: “Episode 4: The Human in the Loop Who Can’t Say No”

Operational AI Readiness, the show. Short episodes, one obstacle each: what goes wrong, what good looks like, and one thing you can do this week. Episode 4 of 12.

Podcast and video: coming soon. Subscribe to the free Mercury Brief to hear when this episode is released.

“Don’t worry, there’s a human in the loop.” It may be the most reassuring sentence in AI governance. It appears in vendor pitches, board papers and policies, and it is meant to end the conversation. This episode argues it should start one. A human who sees too little, too late, with no real power to stop anything, is not oversight. They are a witness, and often the person who will be blamed afterwards.

What goes wrong: the seat in Tempe

The example is not an office AI system, and that is deliberate. It is one of the most thoroughly documented cases we have of a human placed in the loop as the final safeguard, and of what happened to that safeguard in practice.

At 9:58 p.m. on 18 March 2018, a test vehicle run by Uber’s Advanced Technologies Group was driving itself on a test loop in Tempe, Arizona, at 45 mph. A woman was crossing the road outside a crosswalk, pushing a bicycle. According to the US National Transportation Safety Board, the automated driving system detected her 5.6 seconds before impact, but never correctly classified her as a pedestrian or predicted her path (National Transportation Safety Board [NTSB], 2019, p. v). It alternated between calling her a vehicle, a bicycle and “other” (p. 17).

Only 1.2 seconds before impact did the system recognize an emergency. It then entered a one-second “action suppression” period, holding back from braking while it checked the hazard or waited for the operator to take over. Uber said it had built that pause because of concerns about false alarms. No alert was given to the operator during it (pp. 13, 17). The design did not allow emergency braking, and relied instead on the operator to intervene (p. v). The Volvo’s own forward collision warning and automatic emergency braking had been switched off during testing (Finding 7, p. 57).

The operator was looking down toward the centre console, where she had placed her phone. She looked up about one second before impact and began steering 0.02 seconds before the car struck the pedestrian, at 39 mph. The pedestrian died (p. v).

The NTSB, which does not assign fault or blame, found the probable cause was the operator’s failure to monitor the road because she was distracted by her phone. It also found that, attentive, she would likely have had time to avoid or reduce the crash (Finding 9, p. 57). But its contributing causes were about the company: inadequate safety risk assessment, ineffective oversight of vehicle operators, and no adequate mechanisms for addressing operators’ “automation complacency”, all consequences of an inadequate safety culture (p. 59). In September and October 2017, Uber had merged the jobs of two operators into one, which it said let it run more test vehicles (p. 29). It had cameras on operators, but supervisors “rarely reviewed” the videos (p. 44; Finding 13, p. 58).

On paper there was a human in the loop. In practice, the loop was built so that she was very unlikely to close it in time.

Why it happens

Chapter 5 of Operational AI Readiness argues that most bad oversight designs fail for one of four reasons: the review happens too late, the reviewer sees too little, the reviewer has too little authority, or review becomes so rushed and habitual that it turns into ceremony (Bakos, 2026, p. 40). The Tempe case has all four. The useful moment was five seconds out; the human was asked to act with almost none left. The system’s uncertainty never reached her. The design made her the only brake without the alert that would make that realistic. And complacency was a known risk that nobody had built a countermeasure for.

In an office, the same pattern looks like a reviewer who samples cases after the payment has gone out, who sees a polished recommendation but not the sources or flags behind it, and whose screen offers “accept” or “escalate to manager” under heavy queue pressure. Without real authority, the book says, oversight becomes “moral theater performed by people who will still be blamed later” (Bakos, 2026, p. 41).

Key idea: Automation complacency (or automation bias)

When a system is usually right, the people checking it gradually stop checking. Queues get longer, outputs sound confident, and review quietly becomes a click. This is not a character flaw; it is what happens to ordinary people, so oversight has to be designed for it. The EU AI Act names the risk directly, asking that people overseeing high-risk AI stay aware of the tendency of “automatically relying or over-relying on the output” (Regulation (EU) 2024/1689, art. 14(4)(b)). For you, it means “a person reviewed it” is only reassuring if that person was set up to disagree.

What good looks like

Strong oversight designs share a few traits, set out in Chapter 5 (Bakos, 2026, pp. 43–44):

  • In front of the consequence. Review sits where action becomes hard to reverse: before payment release, rejection or access revocation.
  • Context, not just conclusions. Reviewers see sources, flags, uncertainty and the reason the case reached them.
  • Real choices. Reject, revise, request evidence, escalate, pause, with no social penalty for saying no.
  • Disagreement on the record. Every override is recorded with a short reason and reviewed for patterns. As the readiness assessment puts it, falling override rates need an explanation.
  • Designed for complacency. Human review is reserved for cases that need it, reviewers are trained on where the system is weak, and attention is monitored rather than assumed.

The NTSB report contains its own mirror. Before resuming testing in December 2018, Uber put a second operator back in the car, added real-time monitoring of operator attentiveness, and kept the Volvo’s collision warning and emergency braking active. The NTSB said these changes began to address the oversight deficiencies (Findings 8 and 14, pp. 57–58). Less load, monitored attention, a backup brake: none of it exotic, all of it design.

One thing to do this week

Choose one workflow where your organization says “a human reviews it”. Sit beside the reviewer for thirty minutes and write down four answers: when the review happens relative to the consequence, what they see, what they can actually do (list the buttons), and what is recorded when they disagree. Then pull last month’s override count. If it is zero, or nobody knows, you have found your starting point.

What the record does not show

The NTSB investigated a developmental self-driving system, not a decision-support tool, and we are using its findings as an illustration of oversight design rather than as evidence about any office AI product. The report does not quantify how much each contributing factor mattered, and it does not, by its own rules, assign legal fault. We have not reviewed any later legal proceedings, and we have not named the pedestrian or the operator, neither of whom the report names.

Questions you can ask

  • If a decision about you was “reviewed by a person”: Who reviewed it, what information did they have, was an automated or AI system involved, and could they have decided differently? How do I appeal?
  • If you run AI at work: For each place we say a human reviews AI output, does the review happen before the consequence, and what are the reviewer’s actual options?
  • If you run AI at work: How many overrides did our reviewers record last month, and who looked at them?
  • If you sit on a board or audit committee: Ask to see one human-in-the-loop specification: trigger, reviewer, what they see, allowed actions, record, escalation.

Go further. Start with the free Readiness Assessment: pick one live AI workflow and answer the three domain 3 questions on human oversight honestly, counting a “yes” only if you could show the record today. Then read Chapter 5, “Human Oversight That Actually Works”, and Appendix C’s one-page human-in-the-loop specification in Operational AI Readiness, free to read. The first ten readers who send feedback on the book can email their own questions to info@fairbydesign.org, and the expanded Premium edition is free for the first ten with code FIRST10PREMIUM.


FairByDesign is reader-funded. If this helped, get the free Mercury Brief or see ways to support. Support never buys editorial control.

References

Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live (Free ed.). FairByDesign. https://fairbydesign.org/operational-ai-readiness/

National Transportation Safety Board. (2019). Collision between vehicle controlled by developmental automated driving system and pedestrian, Tempe, Arizona, March 18, 2018 (Highway Accident Report NTSB/HAR-19/03). https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1903.pdf

Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), art. 14. (2024). Official Journal of the European Union. https://artificialintelligenceact.eu/article/14/

Leave a Reply