The Evidence Contamination Problem

The Evidence Chain, Part 7 of 10. The first six parts were about what AI systems do. From here the series turns to something deeper: what AI systems are doing to the evidence we use to understand what they do.

In August 2025, a legal academic at the University of Sydney was reading a report the Australian government had paid a consulting firm about A$440,000 to write, a review of the system that automatically penalizes people on welfare. Chris Rudge found a book attributed to a law professor that does not exist and, as the Australian Financial Review first reported, a quotation attributed to a Federal Court judgment that the judgment does not contain (Australian Financial Review, 2025). He said he knew instantly that it was either hallucinated by AI or “the world’s best kept secret” (Associated Press, 2025). The firm was Deloitte. A revised report later disclosed that a generative AI tool chain built on Azure OpenAI’s GPT-4o had been used, the department said the substance and recommendations were unchanged, and Deloitte agreed to repay the final instalment of its contract (Associated Press, 2025; Croft, 2025).

Now notice the timing. That report had already been published

as the government’s official assurance review. If Rudge had not happened to read it closely, its invented authority would have sat in the public record, ready to be cited by the next report, summarized by the next AI tool and relied on by the next decision about people on welfare.

The chain nobody recorded

Think about the last report you relied on at work. Where did it come from? Perhaps someone asked an AI tool to summarize three documents, one of which was itself a summary that another tool produced last quarter from meeting notes a transcription model wrote. The report looks authoritative. It may be accurate. But the chain between what anyone actually saw or said and the sentence in front of you now passes through several machines, and nobody recorded the links.

That is the transition I think we are living through without naming it. AI systems increasingly produce documents, summaries, citations, classifications and records, and that material becomes the input for other people and other AI systems. Each step is useful. The chain as a whole is something we have never had to govern before, and the Deloitte case shows what happens when it fails: a document that looks like it sits close to the evidence turns out to sit on nothing at all.

Try something this week. Pick one document you relied on recently for a real decision, whether a report, a briefing, a summary in your inbox or a source in an essay. Follow one important claim in it back to where it came from. Not to the summary of the source, but to the source. Note how many steps it takes, whether any of those steps were AI-generated, and whether you can get there at all. If you are a student, do it with the reference list of your next assignment. Most of the time you will find the original. The interesting moments are the times you cannot, and the times you realize you had never checked.

It is not one firm, and it is not one profession

The Deloitte report was not a one-off. In November 2025, Newfoundland and Labrador’s independent newsroom The Independent found at least four false citations in a C$1.6 million, 526-page health workforce plan that Deloitte had written for the provincial government, including a paper that the researcher it was attributed to said does not exist (Brake, 2025). In the United States, the White House’s May 2025 “Make America Healthy Again” report cited studies that NOTUS found do not appear to exist; the administration said minor citation and formatting errors had been corrected and the substance remained the same (Kennard & Manto, 2025).

Courts are now counting. A public database maintained by the legal researcher Damien Charlotin lists 2,149 court and tribunal decisions, as of 5 October 2026, in which a court found or implied reliance on hallucinated content, more than two-thirds of them in the United States (Charlotin, 2026). In June 2025, England’s High Court warned that freely available generative AI tools are not capable of conducting reliable legal research, in a judgment dealing with one case where 18 of 45 cited authorities did not exist (R (Ayinde) v London Borough of Haringey; Al-Haroun v Qatar National Bank, 2025). And in 2025, two US federal judges withdrew orders containing errors after members of their own chambers had used AI tools in drafting (Weiss, 2025). The contamination had crossed from the documents courts are asked to rely on into the documents courts produce.

Science is not immune. In January 2026, the detection company GPTZero reported more than a hundred hallucinated citations, each checked by a human, across 53 papers accepted at NeurIPS 2025, one of the most selective conferences in machine learning (Shmatko et al., 2026). The conference’s board responded that incorrect references do not necessarily invalidate a paper’s content (Claburn, 2026). That may be true of each paper. It is less reassuring about a literature that future models will train on.

Why this compounds

In 2024, Shumailov and colleagues showed in Nature that generative models trained indiscriminately on the output of earlier models lose the rare parts of the original data first, and drift away from it over generations (Shumailov et al., 2024). Others have shown that the collapse can be avoided if synthetic data accumulates alongside the original rather than replacing it (Gerstgrasser et al., 2024). Read together, the lesson is not that synthetic content is poison. It is that the original has to stay available and identifiable.

The places where that matters most are often the places with the least original material to begin with. In September 2025, MIT Technology Review reported that volunteers estimated 40 to 60 percent of articles in several African-language Wikipedia editions were uncorrected machine translations, and that because Wikipedia is often the largest body of text in these languages, the errors flow into the models that will be used to translate them next (Judah, 2025). For a teenager in Lagos or Nuuk learning online in their own language, the evidence chain is not an abstraction. It is whether the reference they cite for homework was ever true.

Who owns the original

Ask who decides whether originals are kept, and the answer is uncomfortable: the authors and their tools’ vendors, who are paid for the finished document, not the trail behind it.

The people who can intervene are, in practice, whoever happens to check: a law lecturer, a local reporter, opposing counsel. And the consequences land on people who never see the report at all, such as welfare recipients whose penalties a review was meant to assure, patients whose health system a plan was meant to staff, and litigants whose case turned on an order. When the original disappears, the most powerful document in the room becomes the one nobody can check.

The prediction

My prediction is that regulation will move beyond identifying AI-generated content toward protecting the provenance and evidentiary status of the information consequential decisions rely on. The European Union’s transparency rules for AI-generated content began to apply in August 2026, backed by a Commission code of practice on marking and labelling (European Commission, 2026a). Those rules answer whether a machine touched something. The question that matters for a loan decision, a medical record, a court filing or a government review is a different one: what evidence existed before the AI touched it, and can we still get back to it?

For Mercury Intelligence members: the section below walks the Deloitte review through a scale of distance from the original evidence, maps the gap clause by clause across evidence law, the EU AI Act and provenance standards, sets out an evidence-distance audit you can run this quarter, and tests the argument against the view that the system already catches these errors.

---

Support independent investigation. Every case in this part was caught because someone outside the system went back to the original. That is what FairByDesign does, and it only works if we stay independent of the institutions we check. Give once ($5, $10, $25, $50 or any amount), give monthly ($10 or $25), or become a patron: fairbydesign.org/patrons-2.

References

Advisory Committee on Evidence Rules. (2026, May 17). Report of the Advisory Committee on Evidence Rules [Memorandum to the Committee on Rules of Practice and Procedure, June 3–4, 2026 meeting]. Judicial Conference of the United States. https://www.uscourts.gov/sites/default/files/document/advisory_committee_on_evidence_rules_may_2026.pdf

Associated Press. (2025, October 7). Deloitte to partially refund Australian government for report with apparent AI-generated errors. The Gleaner. https://jamaica-gleaner.com/article/news/20251007/deloitte-partially-refund-australian-government-report-apparent-ai-generated

Australian Financial Review. (2025, August 25). Deloitte report suspected of containing AI invented quote. Australian Financial Review. https://www.afr.com/companies/professional-services/deloitte-report-suspected-of-ai-invented-quote-from-robo-debt-case-20250825-p5mpjj

Australian Greens. (2025, October 9). Greens slam Deloitte’s unethical behaviour and measly refund over AI report bungle [Media release]. https://greens.org.au/news/media-release/greens-slam-deloittes-unethical-behaviour-and-measly-refund-over-ai-report

Bansal, A. (2026, February 10). MeitY notifies new amendments to IT Rules to regulate synthetic media and deepfakes. MediaNama. https://www.medianama.com/2026/02/223-meity-amendments-it-rules-synthetic-media-deepfakes/

Brake, J. (2025, November 22). Major N.L. healthcare report contains errors likely generated by A.I. The Independent. https://theindependent.ca/news/lji/major-n-l-healthcare-report-contains-errors-likely-generated-by-a-i/

Brake, J. (2026a, March 7). Accounting watchdog to investigate Deloitte over fake citations in $1.6M healthcare report. The Independent. https://theindependent.ca/news/lji/accounting-watchdog-to-investigate-deloitte-over-fake-citations-in-1-6-million-healthcare-report/

Brake, J. (2026b, August 26). Watchdog says it can’t investigate Deloitte over use of AI in $1.6M government report. The Independent. https://theindependent.ca/news/lji/watchdog-says-it-cant-investigate-deloitte-over-use-of-ai-in-1-6m-government-report/

Charlotin, D. (2026). AI hallucination cases [Database]. Retrieved October 7, 2026, from https://www.damiencharlotin.com/hallucinations/

Claburn, T. (2026, January 22). AI conference’s papers contaminated by AI hallucinations. The Register. https://www.theregister.com/2026/01/22/neurips_papers_contaiminated_ai_hallucinations/

Coalition for Content Provenance and Authenticity. (2025). C2PA technical specification (Version 2.2). https://spec.c2pa.org/specifications/specifications/2.2/specs/C2PA_Specification.html

Croft, D. (2025, October 9). Deloitte to refund government after using AI in $440k report. Accounting Times.

https://www.accountingtimes.com.au/technology/deloitte-to-refund-government-after-using-ai-in-440k-report

European Commission. (2026a, June 10). Code of practice on marking and labelling of AI-generated content. https://digital-strategy.ec.europa.eu/en/policies/code-practice-ai-generated-content

European Commission. (2026b, July 20). Guidelines on transparency obligations for providers and deployers of AI systems. https://digital-strategy.ec.europa.eu/en/library/guidelines-transparency-obligations-providers-and-deployers-ai-systems

Fed. R. Evid. 901. Authenticating or identifying evidence. https://www.law.cornell.edu/rules/fre/rule_901

Fed. R. Evid. 902. Evidence that is self-authenticating. https://www.law.cornell.edu/rules/fre/rule_902

Gerstgrasser, M., Schaeffer, R., Dey, A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T., Agrawal, R., Pai, D., Gromov, A., Roberts, D. A., Yang, D., Donoho, D. L., & Koyejo, S. (2024). Is model collapse inevitable? Breaking the curse of recursion by accumulating real and synthetic data (arXiv:2404.01413). arXiv. https://arxiv.org/abs/2404.01413

Hunton Andrews Kurth. (2026, July 28). EU Digital Omnibus on AI enters into force. Privacy & Cybersecurity Law Blog. https://www.hunton.com/privacy-and-cybersecurity-law-blog/eu-digital-omnibus-on-ai-enters-into-force

Judah, J. (2025, September 25). How AI and Wikipedia have sent vulnerable languages into a doom spiral. MIT Technology Review. https://www.technologyreview.com/2025/09/25/1124005/ai-wikipedia-vulnerable-languages-doom-spiral/

Kennard, E., & Manto, M. (2025, May 29). The MAHA report cites studies that don’t exist. NOTUS. https://www.notus.org/health-science/make-america-healthy-again-report-citation-errors

Kye, E., & Troutman Privacy Team. (2025, October 28). California AI Transparency Act amendments signed into law. Troutman Privacy. https://www.troutmanprivacy.com/2025/10/california-ai-transparency-act-amendments-signed-into-law/

National Institute of Standards and Technology. (2024). Reducing risks posed by synthetic content: An overview of technical approaches to digital content transparency (NIST AI 100-4). U.S. Department of Commerce. https://doi.org/10.6028/NIST.AI.100-4

R (Ayinde) v London Borough of Haringey; Al-Haroun v Qatar National Bank QPSC [2025] EWHC 1383 (Admin). https://caselaw.nationalarchives.gov.uk/ewhc/admin/2025/1383

Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (2024). Official Journal of the European Union, L, 2024/1689. http://data.europa.eu/eli/reg/2024/1689/oj

Shmatko, N., Adam, A., Esau, P., Cui, A., & Tian, E. (2026, January 21). GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers. GPTZero. https://gptzero.me/news/neurips/

Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631(8022), 755–759. https://doi.org/10.1038/s41586-024-07566-y

Weiss, D. C. (2025, October 24). 2 federal judges reveal AI use by staff members led to error-riddled opinions. ABA Journal. https://www.abajournal.com/news/article/2-federal-judges-reveal-ai-use-by-staff-members-led-to-error-riddled-opinions

Williams, M. (2026, July 29). Machine-generated evidence challenges the federal rules: Inside the contested proposed Federal Rule of Evidence 707. Proskauer Rose LLP. https://www.proskauer.com/blog/machine-generated-evidence-challenges-the-federal-rules-inside-the-contested-proposed-federal-rule-of-evidence-707

Govern retrieval sources as dependencies: Chapter 7, fifth rule, of Operational AI Readiness (free).