Picture an ordinary appointment. Your doctor asks whether you mind if a small app listens in, explains that it writes up the visit so they can look at you instead of a keyboard, and you say yes. Twenty minutes later there is a tidy clinical note in your record. You will probably never read it. The next clinician who opens your file almost certainly will.
In Ontario, the province set out to make that kind of tool easier for doctors to buy. Supply Ontario, the provincial procurement agency, ran a competition and approved 20 AI scribe systems that physicians, nurse practitioners, therapists and other health professionals can purchase from a pre-qualified list (Office of the Auditor General of Ontario [OAGO], 2026, p. 20). In May, Ontario’s Auditor General, Shelley Spence, published a special report on the government’s use of artificial intelligence (Casey, 2026). Its pages on those scribes are some of the most useful public evidence we have seen about how an AI tool gets approved, and about what the word “approved” can quietly leave out.
What the testing found
As part of the procurement, Supply Ontario gave every bidding vendor the same two simulated recordings of a health-care professional talking with a patient. The vendors ran them through their systems and submitted the notes, which medical professionals from OntarioMD and Ontario Health then assessed for accuracy and completeness (OAGO, 2026, p. 22).
The Auditor General reviewed the evaluators’ comments for the 20 systems that went on to be approved. All 20 showed at least one type of inaccuracy (OAGO, 2026, p. 23):
- Invented content. Nine of 20 systems fabricated information and added steps to the patient’s treatment plan, such as a therapy referral or blood tests, that were never mentioned in the recording. Evaluators also flagged notes stating “no masses found,” or recording anxiety, when neither had been discussed.
- The wrong medicine. Notes from 12 of 20 systems recorded a different drug than the one the doctor prescribed.
- Missing mental-health details. Notes from 17 of 20 systems missed key details of the patient’s mental-health issues in at least one of the two tests. Six of the 20 missed them, fully or partly, in both.
These systems were approved anyway. To understand how, look at the scorecard. Accuracy of the generated notes was worth 20 of 530 available points, about 4 percent of the total. Having a domestic presence in Ontario was worth 159 points, about 30 percent. No category had a minimum passing score, so the Auditor General notes that a bidder could, in principle, score zero on accuracy, zero on system security and zero on bias controls and still clear the 371-point bar for approval (OAGO, 2026, pp. 21–22).
The test itself was also softer than it looked. Vendors were not asked to run their systems live in front of evaluators. They generated the notes offline and submitted them, with a signed statement that the notes were unedited (OAGO, 2026, p. 28). Security evidence followed a similar pattern: 11 of the 20 approved vendors submitted no third-party audit report, HITRUST certification or ISO 27001 certification, and five did not submit the threat and privacy risk assessments the process required. All of them declared that they maintained the necessary documentation, and they were approved (OAGO, 2026, p. 25). On bias, for example a scribe misunderstanding someone who speaks with an accent, vendors described their processes but were not asked for evidence such as bias-testing results (OAGO, 2026, p. 27).
Key idea: Validation before approval
Validation means testing an AI tool against realistic cases, under conditions you control, before it is allowed into use, and deciding in advance what result counts as a pass. A pass should mean the tool met a stated threshold on the things that could hurt someone, not that it collected enough points elsewhere. For a medical scribe, that means drug names, doses and diagnoses come out right, nothing is invented, and the test is repeated when the product changes. If the tool that writes your health record is approved without that, nobody has actually checked whether it can be trusted with your record.
What “passing” should mean
It is worth being precise about what Ontario did well. It ran a test. It used realistic scenarios, the same ones for every vendor, and it had clinicians mark the results. Many organizations buying AI tools do none of that. The weakness is in what happened next: the test produced clear evidence of errors that matter clinically, and the scoring design meant that evidence could not stop an approval.
In our reading, a validation step only protects people when three things are true. First, the failures that matter most, such as a wrong drug or an invented finding, are named in advance as disqualifying or as triggering a fix-and-retest, rather than being averaged into a total. Second, the test is run in a way that reflects real use, which is why the Auditor General recommended live demonstrations (OAGO, 2026, p. 28). Third, the evidence is kept, so that when a product is updated someone can run the same cases again and compare. Two recordings is a small test suite. It was enough to reveal problems; it is not enough to show their absence.
The missing signature
The government’s main reassurance is that a person stays in the loop. OntarioMD issued guidelines telling doctors to review the system-generated notes manually. But, the Auditor General found, doctors were not required to attest, through a sign-off feature in the scribe software, that they had actually verified a note (OAGO, 2026, p. 24).
That difference sounds small. It is not. A guideline says what should happen. A sign-off record shows what did happen, for this note, by this clinician, at this time. Without it, nobody, including the doctor, can later tell a reviewed note from one that went straight into the chart. And the errors in these tests are exactly the kind a busy reviewer can miss: a plausible drug name, a sensible-sounding referral, a mental-health detail that is simply absent.
The Auditor General contrasted Ontario with England, where NHS England’s April 2025 guidance directed health organizations to use only scribe systems registered with the UK’s medicines regulator as Class I medical devices, and required vendors to meet clinical accuracy requirements (OAGO, 2026, p. 24). In fairness, the UK picture has moved since. In July 2026 the Medicines and Healthcare products Regulatory Agency clarified that a scribe which only transcribes or summarizes for clinician review is not, on that basis alone, a medical device (MHRA, 2026), and according to law firm Covington, NHS England has updated its guidance to match (Cowlishaw & Spivey, 2026). The underlying point survives: England chose to treat the accuracy of these tools as a clinical safety question, with a named regulator, rather than as one line in a purchasing scorecard.
What the government said
Supply Ontario agreed to review guidance from Ontario’s Information and Privacy Commissioner and to “work with vendors to determine the feasibility” of mandatory confirmation that notes were reviewed in future scribe procurements (OAGO, 2026, p. 36). It agreed to require bias-testing results in future AI procurements, said it will “consider” live demonstrations, and has added annual third-party security reports to all current scribe contracts. It disagreed with the recommendation to give security, privacy, bias and accuracy more weight, calling the existing weighting appropriate, while agreeing to set minimum thresholds in future (OAGO, 2026, pp. 35–37).
On the day the report was released, Stephen Crawford, the minister responsible for procurement, said medical practitioners will always review notes before a decision is made, according to The Canadian Press (Casey, 2026). He also stressed, as quoted by Global News and reported by Canadian HR Reporter, that the errors came from testing and were “not actually in operational use with doctors” (Wilson, 2026). That is accurate about where the errors were observed. The question the report leaves open is what has changed in the approved products since.
What the record does not show
This report does not show that any patient was harmed, and we have found no public evidence that one was. The errors were observed in two simulated recordings during procurement, not in real appointments. Vendors may have improved their systems since; the report does not say. Doctors may be catching many of these errors in practice; there is no sign-off record that would tell us either way. The figures describe the 20 approved systems at the time of evaluation, not every AI scribe in use in Ontario, since clinics can also buy tools outside the approved list (OAGO, 2026, p. 20). What the record does show is narrower and still important: the province had clear warning signs in hand and a process that was not built to act on them.
Questions you can ask
- If you are a patient: ask whether your clinic uses an AI scribe and which one, and whether you can say no to being recorded without it affecting your care.
- Also as a patient: ask whether your doctor reads and corrects the note before it is saved, and how to request a copy of your record so you can check medications and diagnoses yourself.
- If you run or buy for a clinic or health organization: ask your vendor for its test results on realistic cases, including drug names and mental-health details, and what it changed after them. Ask whether the software records a clinician sign-off on each note.
- If you procure any AI tool: write down, before the competition starts, which failures disqualify a bidder outright. If accuracy can be outscored by geography, your scorecard has made a decision you never meant to make.
Go further. Operational AI Readiness is FairByDesign’s free field manual for AI that is already live. Chapter 5, Human Oversight That Actually Works, explains why a review only counts when it leaves a record, and Chapter 2 maps where risk appears once a system meets real users. Appendix D gives a vendor due-diligence question bank, and Appendix E a minimal readiness test suite you can adapt before approving any tool. Read it free, or test one workflow with the free readiness assessment.
Follow what happens next. We will be watching whether Ontario follows through on note sign-off and stronger testing. The free Mercury Brief brings you findings like this, with the record behind them, in a short email. Get the free Mercury Brief.
FairByDesign is reader-funded. If this helped, get the free Mercury Brief or see ways to support. Support never buys editorial control.
References
Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live. FairByDesign. https://fairbydesign.org/operational-ai-readiness/
Casey, L. (2026, May 12). Ontario auditor general finds problems with medical AI transcription tool. The Canadian Press via National Newswatch. https://nationalnewswatch.com/2026/05/12/ontario-auditor-general-finds-problems-with-medical-ai-transcription-tool
Cowlishaw, S., & Spivey, D. (2026, August 17). New UK guidance clarifies medical device status of AI scribes. Global Policy Watch (Covington & Burling). https://www.globalpolicywatch.com/2026/08/new-uk-guidance-clarifies-medical-device-status-of-ai-scribes/
Medicines and Healthcare products Regulatory Agency. (2026, July 29). Ambient voice technology-enabled products. GOV.UK. https://www.gov.uk/government/publications/ambient-voice-technology-enabled-products/ambient-voice-technology-enabled-products
Office of the Auditor General of Ontario. (2026). Use of artificial intelligence in the Ontario government [Special report]. https://www.auditor.on.ca/en/content/specialreports/specialreports/en26/2026_AI_EN.pdf
Wilson, J. (2026, May 13). AI systems for Ontario doctors hallucinate: Auditor general. Canadian HR Reporter. https://www.hrreporter.com/focus-areas/automation-ai/ai-systems-for-ontario-doctors-hallucinate-auditor-general/394430
Leave a Reply
You must be logged in to post a comment.