The Human Witness

The Evidence Chain, final part. Nine parts traced one escalation, from rebuilding what an AI did to discovering that the evidence itself is increasingly machine-made. This last part asks what remains at the end of the chain, and what we should build next.

Imagine opening a letter from the government telling you that you owe it thousands of dollars for welfare payments you received years ago. You do not remember earning more than you declared. The letter is confident and the figure is precise. If you want to dispute it, you are told, send in your payslips or bank statements from that period, years back, from jobs you may have left long ago (Henriques-Gomes, 2020).

That happened in Australia, at scale, between 2015 and 2019. The scheme the public came to call robodebt matched annual income data held by the tax office against the fortnightly income people had reported to the welfare agency, and where the two did not line up it spread the annual figure evenly across the year, “averaging” it, and treated the difference as a debt (Henriques-Gomes, 2020). The court that approved a settlement in 2021 heard that the scheme had unlawfully raised $1.76 billion in debts against 443,000 people, and the judge, Justice Bernard Murphy, called it “a shameful chapter in the administration of the commonwealth social security system and a massive failure of public administration” (Henriques-Gomes, 2021). When the royal commission under Catherine Holmes reported in July 2023, it was blunter still: robodebt was “a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals” (Karp, 2023).

There was no artificial intelligence in robodebt, and that is exactly why it belongs at the end of this series. A machine produced an inference about what had happened in people’s lives. An institution treated the inference as the event. And the evidence that could settle the question, a payslip printed on the day, a bank statement, a person’s own account, was held by the people least able to produce it. That is the whole Evidence Chain in one story, and it is about to become much more common.

The evidence collapse

Look back at where this series has been. Part 1 asked whether anyone could reconstruct what an AI system did without taking the operator’s word for it. Since then we have seen approval drift from execution, systems act on interpretations of what people want, those interpretations get poisoned, apparently independent systems share hidden dependencies and fail together, machine output become the evidence future machines learn from, that evidence agree with itself, and reference trails that lead nowhere. Each was a gap between a record and the thing it claims to describe. Put together, they describe something that deserves a name. I call it the evidence collapse.

The evidence collapse is not misinformation, although misinformation lives inside it. It is the loss of evidentiary independence. As AI becomes both an observer and an author of information, our records stop reliably distinguishing observation from inference, primary evidence from derivative copies, independent sources from computational repeats, and records of events from reconstructions of records of events. Research on models trained on the output of earlier models gives the mechanism a precise shape: the rare and unusual parts of the original data disappear first, and the models drift away from what the originals contained (Shumailov et al., 2024). An information ecosystem can do the same. The records get more fluent, more consistent and more numerous, while less and less of them touches the world.

What a human witness is, and is not

This is where the argument usually goes wrong, so I want to be careful. The conclusion is not that people are more accurate than machines. Often they are not. Memory is reconstructive, observers are biased, and much of what AI systems produce is more accurate than the human work it replaces. If this series ended with “trust people, not AI,” it would be wrong and useless.

The claim is narrower. When the chain of evidence becomes recursively machine-mediated, independently situated human observation becomes a scarce form of provenance, and one we should protect on purpose. By a human witness I do not mean only a person in a witness box. I mean any human-originated observation or contemporaneous record that keeps a connection to the event before later machine transformations: an eyewitness account, a field note, an original photograph, a signed professional record, a laboratory notebook, a source interview, a physical count of cash in a drawer, a payslip. What makes it valuable is not that a person is perfect. It is proximity to the event, and the fact that it did not come from the system it is being used to check.

Robodebt shows both edges of that claim, and the second edge matters as much as the first. People’s own payslips and bank statements were what exposed debts that had been wrongly raised. But when the settlement was approved, about 200,000 people originally in the class action received nothing from it, because their debts had eventually been substantiated using their own payslips or other evidence (Henriques-Gomes, 2021). The contemporaneous record cut both ways. It was not on the person’s side or the government’s side. It was on the side of what had actually happened. That is what a witness is for.

Who holds the record holds the power

Every part of this series has ended up as a question about power, and this one is the sharpest. In robodebt the institution held the machine and the people held the paper, and the burden of proof was moved onto the side least equipped to carry it. Justice Murphy noted that many people facing these debts were marginalised or vulnerable and ill-equipped to challenge an overpayment (Henriques-Gomes, 2021), and the royal commission recorded “young people being driven to despair by demands for payment” (Karp, 2023). In the Post Office Horizon scandal in Britain, the pattern ran the other way round: the supplier held the error logs and the subpostmasters held only their own word, and a legal presumption that computers work properly kept the logs out of reach for years (Marshall et al., 2021).

Carry that into the world now being built and ask who will hold the originals. Your messages, photos, meeting notes and health records increasingly pass through systems that summarize, enhance, transcribe and reorganize them before anyone relies on them. If you are young, you will spend your whole working life inside that world, and so will your children. The decisions about which records are kept in their original form, who can see them and who has to produce them in a dispute are being made now, mostly by the people who build and sell the systems. You are entitled to see those decisions while they are being made, not after.

The law is reaching for this, slowly

Legal systems already know the intuition. US federal evidence law admits business records made at or near the time by someone with knowledge, because contemporaneity and knowledge are what tie a record to reality (Fed. R. Evid. 803(6)). But the attempt to hold machine-generated evidence to a reliability standard has stalled: proposed Rule 707 closed for comment in February 2026, and in June the judiciary’s Standing Committee decided not to recommend it at that time, sending it back for further study alongside the problem of deepfakes (Williams, 2026). In England and Wales, the presumption that a computer was working properly, which framed the Horizon prosecutions, is under review: the Ministry of Justice called for evidence in January 2025, the consultation closed that April, and as I write in October 2026 no government response has been published (Ministry of Justice, 2025). The public inquiry into Horizon published a progress update on the rest of its final report in July 2026 (Post Office Horizon IT Inquiry, 2026). Each of these is a legal system reaching, unevenly, for the distinction between a record and evidence of what happened.

The prediction

The final prediction of this series is that, as AI-mediated information becomes pervasive, consequential sectors will increasingly distinguish ordinary digital records from evidence with demonstrable provenance to an underlying human or physical-world observation. I do not expect one grand law. I expect pressure through many channels at once: AI transparency rules, consumer and welfare law, financial supervision, product liability, records management, scientific integrity, litigation and discovery, and the standards for digital identity and provenance. Human-origin evidence will become part of our epistemic infrastructure, something we build, fund and protect deliberately, the way we already protect land registries, birth records and the chain of custody for physical evidence.

That changes the question this series started with. The final question is not “can we prove what the AI did?” It is “can we still prove what actually happened before the machines began telling us what happened?”

For Mercury Intelligence members: the section below is the full working framework that ties the series together: the eight questions to ask of any evidence, robodebt walked through all ten layers of the chain, the rules mapped clause by clause, the ten-layer audit, the strongest objection, what would confirm or refute the prediction, and the research agenda that comes next.

---

Support independent investigation. This series ends where next year’s work begins: testing whether the controls, overseers and records we are promised actually hold. That work has to be independent of the people who build and sell the systems, or it is worth nothing. Give once ($5, $10, $25, $50 or any amount), give monthly ($10 or $25), or become a patron: fairbydesign.org/patrons-2.

References

Chan, S., Pataranutaporn, P., Suri, A., Zulfikar, W., Maes, P., & Loftus, E. F. (2024). Conversational AI powered by large language models amplifies false memories in witness interviews (arXiv:2408.04681) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2408.04681

Fed. R. Evid. 602. https://www.law.cornell.edu/rules/fre/rule_602

Fed. R. Evid. 803(6). https://www.law.cornell.edu/rules/fre/rule_803

Fed. R. Evid. 901. https://www.law.cornell.edu/rules/fre/rule_901

Fed. R. Evid. 902. https://www.law.cornell.edu/rules/fre/rule_902

Henriques-Gomes, L. (2020, May 29). Robodebt: Government to refund 470,000 unlawful Centrelink debts worth $721m. The Guardian. https://www.theguardian.com/australia-news/2020/may/29/robodebt-government-to-repay-470000-unlawful-centrelink-debts-worth-721m

Henriques-Gomes, L. (2021, June 11). Robodebt: Court approves $1.8bn settlement for victims of government’s ‘shameful’ failure. The Guardian. https://www.theguardian.com/australia-news/2021/jun/11/robodebt-court-approves-18bn-settlement-for-victims-of-governments-shameful-failure

Karp, P. (2023, July 7). ‘Crude and cruel’ scheme: Robodebt royal commission report recommends civil and criminal prosecutions. The Guardian. https://www.theguardian.com/australia-news/2023/jul/07/robodebt-royal-commission-final-report-recommends-civil-criminal-prosecutions

Loftus, E. F., & Palmer, J. C. (1974). Reconstruction of automobile destruction: An example of the interaction between language and memory. Journal of Verbal Learning and Verbal Behavior, 13(5), 585–589. https://doi.org/10.1016/S0022-5371(74)80011-3

Marshall, P., Christie, J., Ladkin, P. B., Littlewood, B., Mason, S., Newby, M., Rogers, J., Thimbleby, H., & Thomas, M. (2021). Recommendations for the probity of computer evidence. Digital Evidence and Electronic Signature Law Review, 18, 18–26. https://doi.org/10.14296/deeslr.v18i0.5240

Ministry of Justice. (2025). Use of evidence generated by software in criminal proceedings [Call for evidence, closed 15 April 2025]. GOV.UK. https://www.gov.uk/government/calls-for-evidence/use-of-evidence-generated-by-software-in-criminal-proceedings

National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0) (NIST AI 100-1). U.S. Department of Commerce. https://doi.org/10.6028/NIST.AI.100-1

National Research Council. (2014). Identifying the culprit: Assessing eyewitness identification. The National Academies Press. https://doi.org/10.17226/18891

Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055

Post Office Horizon IT Inquiry. (2026, July 8). Progress update on the remainder of the final report. https://www.postofficehorizoninquiry.org.uk/reports-and-statements

Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). (2016). Official Journal of the European Union, L 119, 1–88. http://data.europa.eu/eli/reg/2016/679/oj

Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (2024). Official Journal of the European Union, L, 2024/1689. http://data.europa.eu/eli/reg/2024/1689/oj

Regulation (EU) 2026/1744 of the European Parliament and of the Council of 8 July 2026 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (Digital Omnibus on AI). (2026). Official Journal of the European Union, L, 2026/1744. http://data.europa.eu/eli/reg/2026/1744/oj

Regulation (EU) No 910/2014 of the European Parliament and of the Council of 23 July 2014 on electronic identification and trust services for electronic transactions in the internal market (eIDAS). (2014). Official Journal of the European Union, L 257, 73–114. http://data.europa.eu/eli/reg/2014/910/oj

Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631(8022), 755–759. https://doi.org/10.1038/s41586-024-07566-y

Wells, G. L., Kovera, M. B., Douglass, A. B., Brewer, N., Meissner, C. A., & Wixted, J. T. (2020). Policy and procedure recommendations for the collection and preservation of eyewitness identification evidence. Law and Human Behavior, 44(1), 3–36. https://doi.org/10.1037/lhb0000359

Williams, M. (2026, July 29). Machine-generated evidence challenges the Federal Rules: Inside the contested proposed Federal Rule of Evidence 707. Minding Your Business. Proskauer Rose LLP. https://www.proskauer.com/blog/machine-generated-evidence-challenges-the-federal-rules-inside-the-contested-proposed-federal-rule-of-evidence-707

Wixted, J. T., & Wells, G. L. (2017). The relationship between eyewitness confidence and identification accuracy: A new synthesis. Psychological Science in the Public Interest, 18(1), 10–65. https://doi.org/10.1177/1529100616686966

The practitioner’s version of this whole series is free: Operational AI Readiness.