Ten Minutes, 24 Questions: Would These Three AI Systems Have Passed?

Three AI failures now sit in the public record. In Ontario, AI medical scribes approved for use by doctors wrote down the wrong drug in testing. In Australia, a fraud-detection model was switched on for Medicare compliance work while the people responsible for cyber security did not know it existed. And in Philadelphia, an AI model being tested by its developer submitted a tip about an unsolved murder to a police tip line, and the police heard about it months later.

These are different systems, in different countries, built by different people. But each of them failed a question that anyone responsible for an AI workflow could ask in about ten minutes. We know, because those questions are written down in FairByDesign’s free Operational AI Readiness Assessment, and this post walks all three cases through it. Then it shows you how to run it yourself, and how the same questions work for anyone on the receiving end of an AI decision.

Three cases, one set of questions

A ground rule first. What follows is based on the public record, not an audit. We have not seen inside any of these systems. We are asking: on what the auditors, the developer and the police have published, could the people responsible have answered “Yes, and we could show the record today”?

Ontario’s AI scribes. Ontario’s Auditor General reviewed how Supply Ontario chose the AI scribe systems that health-care providers can buy. In the procurement tests, every vendor received the same two simulated doctor–patient recordings. Evaluators found that notes from 12 of the 20 approved systems captured a different drug than the one prescribed, and that 9 of the 20 fabricated information, such as treatment steps that were never discussed (Office of the Auditor General of Ontario, 2026, p. 23). All 20 were approved anyway. Doctors were given guidelines to review the notes, but they were not required to attest, through a sign-off feature, that they had actually done so (p. 24). Assessment question 3.1 asks whether “human review happen[s] before a consequential action becomes hard to reverse.” Maybe it does, in most clinics. The record shows no way to prove it.

Australia’s Medicare fraud model. The Australian National Audit Office examined how the Department of Health, Disability and Ageing used AI to detect health providers’ non-compliance with Medicare (ANAO, 2026). The audit found that business approval before deployment, and for changes, was not consistently captured: “In some cases, approval was provided verbally only” (ANAO, 2026, Table 4.3). Teams responsible for cyber security were unaware of the fraud detection system’s operation, and a cyber security risk assessment had not been completed (Table 4.1). The use case assessment that rated the system as low ethical risk “was not recorded in DHDA’s AI use case register” (para. 4.8). The AI model was replaced with a standard statistical model in December 2025 (para. 1.26), and the department agreed to all three of the audit’s recommendations.

The false police tip. Anthropic has published that one of its models, Claude Haiku 4.5, was generating and performing example tasks on randomly selected live webpages as part of an evaluation. It landed on a page about an unsolved homicide, filled in the police department’s tip form with name and contact fields left empty, and submitted it. The model had been told never to log in, enter personal data or submit anything destructive, “but the instructions did not rule out form submissions” (Anthropic, 2026). The tip went to spam and was never forwarded for investigation. According to NBC10, police say the tip was posted on 18 July; Anthropic said it found the incident on 28 September; and police say they were notified on 7 October, a delay they called “unacceptable” (Chang, 2026). Anthropic gives 8 October as the date it shared the finding with the department.

Case	Assessment question	What the public record shows
Ontario AI scribes	3.1 Does human review happen before a consequential action becomes hard to reverse?	Doctors were not required to attest they had verified notes.
Ontario AI scribes	8.1 Does your test suite go beyond happy-path cases?	Tests found wrong drugs in 12 of 20 systems and fabrications in 9; all 20 approved.
Ontario AI scribes	5.3 Is each approval decision recorded with a short rationale?	No sign-off record that a clinician approved each note.
Medicare fraud model	1.1 Is this workflow recorded in an inventory?	Use case assessment not recorded in the AI use case register (para. 4.8).
Medicare fraud model	2.2 / 2.3 Are approver roles named and documented? Who can stop it?	Approval not consistently captured; in some cases verbal only (Table 4.3).
Medicare fraud model	Domain 7, security boundaries	Cyber security teams unaware; no cyber security risk assessment (Table 4.1).
False police tip	7.2 Are tools and permissions limited to an approved, least-privilege set?	Instructions did not rule out form submissions on live sites.
False police tip	8.3 Is it written down who is notified and what evidence is preserved?	About ten weeks from tip to discovery; police told about nine days later.

Why the tool exists: having a control versus proving it

The assessment has one rule that does most of the work: “Yes” only counts in full if you could show the record today. Every control question offers three answers: “Yes, and we could show the record today”, “Partly, or yes but we couldn’t show it”, and “No / we don’t know”. The middle answer is where most organizations honestly live, and the tool is designed to make that visible rather than flattering.

Before the control questions, the tool asks five classification questions from Chapter 3 of the book, to decide whether the workflow is consequential at all. They ask whether it influences a decision or action; whether that influence could “materially affect money, access, employment, eligibility, security, legal position, safety or rights”; whether it can trigger or speed up downstream actions; whether people will foreseeably over-rely on it; and whether you would need to reconstruct what happened to explain yourself. “Not sure” counts as yes. Uncertainty about whether a system matters is a reason to treat it as if it does.

All three of our cases are consequential by that test. Medical notes shape treatment. Fraud flags shape who gets investigated. A police tip can shape who gets questioned. The overall result is set by your weakest domain, because, as the tool puts it, “a strong total cannot compensate for a missing control.” A brilliant model with no record of who approved it is still a system you cannot defend.

How to use it, step by step

1. Pick one live workflow. Not your AI programme, not your vendor. One workflow that is running today, such as triage notes, claims flags or a support agent with tool access.
2. Put three people in the room. The person who owns the workflow, someone who reviews its outputs, and someone from security. The Medicare case shows why the third seat matters.
3. Answer honestly. If you would need to go and look for the record, the answer is “Partly.” If nobody knows, it is “No / we don’t know.”
4. Read your weakest domain first. The result labels each domain “Active, not ready”, “Described, not demonstrated” or “Legible under pressure”. Your overall label is your weakest one.
5. Take the three next moves. The tool gives you three concrete actions, such as naming one person who can stop the workflow, or writing a one-page “what happens if it misbehaves”.
6. Read the named chapter. Each move points to the chapter of Operational AI Readiness that teaches it. The Standard edition is free.
7. Repeat in 30 days. Chapter 10 sets out a 30-day readiness sprint built to produce records, not statements of intent. Run the assessment again at the end and compare.

Nothing you answer is sent anywhere or stored. It is a self-assessment, not legal advice, an audit or a certification.

Questions anyone can ask

You do not need to run an AI system to use these questions. If a clinic, benefits office, school or employer uses AI in decisions about you, the same ideas translate into plain language:

• Who can stop it? Is there a named person who can pause this system if it goes wrong? (Question 2.3)
• Will a person review this before it affects me? Not afterwards, as a spot check, but before? (Question 3.1)
• What gets recorded, and can I see it? If I challenge a decision, can you show what the system produced and who signed it off? (Domain 4)
• Who approved using AI for this? Was that decision written down? (Question 5.3)
• What happens if it gets something wrong? Who will tell me, and how quickly? (Question 8.3)

For organizations, the matching question is simpler and harder: if a patient, citizen or worker asked you those five things tomorrow, could you answer from records rather than reassurance?

The eight domains in one minute

• 1. Inventory and scope: is the workflow written down, with its purpose and excluded uses? (Chapters 3 and 4)
• 2. Ownership and authority: one named owner, named approvers and someone who can stop it. (Chapters 4 and 5)
• 3. Human oversight that works: review before harm, with context, and recorded overrides. (Chapter 5)
• 4. The evidence spine: can you rebuild one real case end to end from records? (Chapter 6)
• 5. Approval gates for high-impact actions: explicit triggers and recorded rationale. (Chapter 9)
• 6. Change control beyond the model: prompts, tools and retrieval changes logged and tested, not just model swaps. (Chapter 8)
• 7. LLM and agent security boundaries: untrusted content kept apart from instructions, tools limited to least privilege. (Chapter 7)
• 8. Validation and failure readiness: tests beyond the happy path, a rehearsed pause and a written notification plan. (Chapters 4, 7 and 10)

What the record does not show

We are reading public documents, not system logs. The Ontario findings come from procurement tests on simulated recordings, not from audits of real patient notes, and the Auditor General did not report patient harm. The Medicare audit describes governance gaps, not proven harm to any provider or patient, and the ANAO reports that process improvements are planned and underway. In the Philadelphia case the tip went to spam and, according to police, was never acted on, and Anthropic has described changes since, including restricting its models’ web tools and blocking live internet access in internal evaluations pending checks. Some of these organizations may hold records the public cannot see. The assessment cannot assign blame; it can show which questions should have had a recorded answer before anything went live.

Your next step

Pick one workflow, book ten minutes with its owner, and answer honestly. Then read the chapter it points you to.

Know someone who owns an AI workflow at work? Send them this post. The questions are easier to answer before anyone else asks them.

---

FairByDesign is reader-funded. If this helped, get the free Mercury Brief or see ways to support. Support never buys editorial control.

References

Anthropic. (2026, October 9). Investigating unintended model actions in our evaluations and internal use. https://www.anthropic.com/research/investigating-unintended-model-actions

Australian National Audit Office. (2026, September 21). Artificial intelligence and Medicare benefits integrity (Auditor-General Report No. 5 2026–27). https://www.anao.gov.au/work/performance-audit/artificial-intelligence-and-medicare-benefits-integrity

Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live. FairByDesign. https://fairbydesign.org/operational-ai-readiness/

Chang, D. (2026, October 9). Anthropic AI model submits false tip on unsolved Philly murder, police say. NBC10 Philadelphia. https://www.nbcphiladelphia.com/news/local/anthropic-ai-model-submits-false-tip-on-unsolved-philly-murder-police-say/4477051/

FairByDesign. (2026). Operational AI readiness assessment [Online self-assessment tool]. https://fairbydesign.org/readiness-assessment/

Office of the Auditor General of Ontario. (2026). Use of artificial intelligence in the Ontario government [Special report]. https://www.auditor.on.ca/en/content/specialreports/specialreports/en26/2026_AI_EN.pdf