Approved Out Loud: The Medicare Fraud AI With No Paper Trail for Its Sign-Off

Between July 2024 and December 2025, a piece of software was helping decide which doctors, pharmacists and other health providers deserved a closer look for possible Medicare fraud. It was one of six models in a detection system run by the Department of Health, Disability and Ageing, and it was the only one that used AI (Australian National Audit Office [ANAO], 2026, para. 1.26). On 21 September, Australia’s national auditor published what it found when it went looking for the paperwork behind it.

One line in that report deserves to be read slowly. The department told the auditor that business approval before deployment, and for later changes, “was not consistently captured. In some cases, approval was provided verbally only” (ANAO, 2026, Table 4.3).

Approved out loud. For a system pointing a government fraud team at named professionals, that is the detail I keep coming back to.

What the system did

The stakes are large. Medicare, the Pharmaceutical Benefits Scheme and the Child Dental Benefits Schedule paid out more than $52 billion in 2024–25, and a 2023 review cited by the auditor estimated provider non-compliance at $1.5 to $3 billion (ANAO, 2026, para. 7). The department had a real problem: its rules for spotting suspicious billing were flagging more providers than staff could check by hand, so it built an AI model to help focus on the riskier ones (ANAO, 2026, para. 5.9).

The results were mixed in a way worth understanding properly. Between August 2023 and August 2025, the AI model raised 597 potential matters. Staff reviewed 82 of them, about 14 per cent. Of those 82, 18 were judged true positives and 64 false positives, a true positive rate of 22 per cent (ANAO, 2026, Table 5.1). Put plainly: of the flagged cases staff actually checked, about one in five held up. That is not the same as saying the model was wrong most of the time, because more than 500 flags were never reviewed at all, and the department told the auditor that rates should not be compared across models hunting for different behaviours (ANAO, 2026, Table 5.1, note c).

Eight matters were referred for preliminary analysis using, in part, the AI model. By February 2026, five had been confirmed as potential fraud or non-compliance and passed on for investigation or audit, and no money had yet been recovered (ANAO, 2026, para. 5.5). Then, in December 2025, the department stopped using the model, because the people needed to assess its “signals” by hand could not keep up with the volume. The auditor found no assessment of the benefits and costs of that decision, and no analysis of what the system had cost to build (ANAO, 2026, paras. 18–19, 5.13). The department has said it intends to use AI models in the system again (ANAO, 2026, para. 5.14).

The approval that was not written down

The auditor rated the model’s “business validation and approval” as not aligned with better practice, its lowest mark. Evidence that business users had verified and validated the model’s outputs was not consistently captured, and neither was approval to deploy it or change it (ANAO, 2026, Table 4.3). On the technical side the picture was better but not clean: technical approval for the first deployment was recorded in the code release system, yet the department later found changes to models had been deployed without corresponding technical approval or testing, and system settings forcing approval before deployment were not fully in place until early 2026 (ANAO, 2026, Table 4.3).

Why does this matter to anyone outside a government IT department? Because being selected for a fraud review is not a neutral event. For a GP, a physiotherapist or a small pharmacy, it can mean months of stress, legal costs and a cloud over a practice. If a model helps choose who gets that letter, then somebody should be able to say, on paper, who decided the model was fit for that job, what version they approved, and what evidence they looked at. And the data flowing through it was patients’ and providers’ claiming records, personal information about real people.

The system security did not know about

The second finding is quieter but, to my mind, just as serious. The department told the auditor in March 2026 that the fraud detection system had been built before its enterprise AI governance existed and had not been managed within its governance and security framework. “Teams responsible for cyber security were unaware of the fraud detection system’s operation,” and no cyber security risk assessment had been done (ANAO, 2026, Table 4.1).

More broadly, the auditor found no considered assessment before development and deployment of ethical risks, legal risks, data suitability, sensitivity, privacy or cyber security (ANAO, 2026, para. 16). A November 2024 assessment rated the system as low inherent ethical risk despite its use of personal information, which meant privacy and legal risks were not examined in more detail, no legal advice was sought, and the assessment never made it into the department’s AI use case register. In May 2026 a privacy threshold assessment identified high-risk factors, linked to using personal information to profile or predict people’s behaviour (ANAO, 2026, para. 4.8; Table 4.1).

Here is the security principle hiding in that finding: you cannot protect what you do not know you have. Security teams work from an inventory. If an AI system holding sensitive health claims data is not on it, it is not in their threat models, their patching, their access reviews or their incident plans. And this was not an isolated gap. In April 2026 the department found that impact assessments for 22 in-scope AI use cases had not been appropriately approved, and a stocktake in June 2026 found 10 unapproved AI tools in use (ANAO, 2026, para. 4.9).

What the department got right

Fairness matters here, and the report is more balanced than a single quote suggests. The auditor judged the department’s enterprise-level arrangements for managing AI “largely effective”; it was the arrangements for AI in provider compliance that were “partly effective” (ANAO, 2026, para. 10). The fraud system logged its inputs and code changes, used version control and could reproduce its outputs, which earned the auditor’s highest rating for auditability (ANAO, 2026, Table 4.1). The model’s flags went to people for manual assessment rather than straight to action. New change management and governance frameworks from February and July 2026 now set expectations for recording business approval (ANAO, 2026, Table 4.3). And the department agreed to all three recommendations, including one to ensure that “all artificial intelligence systems in use in the department have appropriate authorisation” and that security risks are identified, documented, managed and accepted before systems are authorised for use (ANAO, 2026, para. 4.10).

That recommendation is the whole lesson in one sentence. The department’s own response says future use cases will proceed only once all required approvals are “obtained and recorded” (ANAO, 2026, para. 4.12). Recorded is the word that matters.

What the record does not show

The report does not say who gave the verbal approvals, how many decisions were made that way, or which changes they covered. It does not show that any provider was wrongly investigated or penalised because of the AI model, and it should not be read that way: the eight referred matters drew on several models together, and the auditor notes the department’s reporting did not separate the AI model’s contribution (ANAO, 2026, para. 5.6). We also do not know what happened to the 515 flags that were never reviewed, or whether anyone outside the department was ever told their claims had been scored by AI. Those are fair questions for the department, not conclusions we can draw.

Questions you can ask

• If you are a health provider or patient: when an organization reviews your claims or records, ask whether an automated or AI tool contributed to selecting you, and what human review followed.
• If you run or oversee an AI system: pick one model that affects people and ask for its authorization record. Can you show who approved the version running today, when, and on what evidence?
• If you work in security: ask for the list of every AI system that touches personal data, and check whether each one has a security risk assessment with a named owner.
• If you sit on a board or audit committee: ask how many AI tools a stocktake would find that are not on the official register. Then ask for the stocktake.

If you answer for AI at work, this is the kind of finding your own auditors will ask about next. Mercury Intelligence ($10 a month) gives you our full investigations and a governance impact analysis of each one: what the finding means for your controls, the evidence a reviewer will expect, and the questions to put to your team this quarter. See what is included, or save with the annual plan.

---

FairByDesign is reader-funded. If this helped, get the free Mercury Brief or see ways to support. Support never buys editorial control.

References

Australian National Audit Office. (2026, September 21). Artificial intelligence and Medicare benefits integrity (Auditor-General Report No. 5 of 2026–27). Commonwealth of Australia. https://www.anao.gov.au/work/performance-audit/artificial-intelligence-and-medicare-benefits-integrity

Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live. FairByDesign. https://fairbydesign.org/operational-ai-readiness/