What Evidence Should an Organization Keep About Its AI?

Keep everything and you drown, and may break privacy law. Keep too little and you cannot explain a single decision. The answer is a short list of fields, chosen on purpose.

FairByDesign Operational AI Readiness title card: “What Evidence Should an Organization Keep About Its AI?”

The Control File

Picture a small business owner (an illustrative scene, not a real case). An AI tool helps her firm sort customer requests, and a loyal customer has just been refused something he was sure he qualified for. He asks the fair question: why? She opens the system and cannot answer.

There are two ways to fail at AI record-keeping, and many organizations manage both at once. The first is to keep almost nothing that connects an AI output to the decision it shaped, so that when someone asks “why did this happen?” the honest answer is a shrug. The second is to keep enormous volumes of everything, raw prompts, full transcripts and every intermediate token, indefinitely, in a bucket nobody can search, which creates a privacy liability without creating an explanation. Both feel like they are handling the problem. Neither does.

My proposition: the evidence that protects you, and the person on the other end of the decision, is a short list of facts chosen on purpose, not a bigger pile. When the record fails, the customer pays first, because nobody can tell him why. Then the business pays, because it cannot defend what it cannot explain.

What the rulebooks actually ask

It is less than people fear and more than most do. Under the EU AI Act, high-risk systems must be able to record events automatically, and that logging should support three purposes: identifying situations where the system may present a risk, supporting the provider’s post-market monitoring, and supporting the deployer’s monitoring of the system in operation (Regulation (EU) 2024/1689, art. 12(2)). Deployers must keep the logs that are under their control “for a period appropriate to the intended purpose” of the system, “of at least six months,” unless other law says otherwise (art. 26(6)).

The United Kingdom’s voluntary Code of Practice for the cyber security of AI is plainer still: system operators “shall log system and user actions to support security compliance, incident investigations, and vulnerability remediation” (Department for Science, Innovation and Technology, 2025, provision 12.1). So the direction is clear. Logs exist to make monitoring and investigation possible, and there is a floor on how long to keep them.

The catch: keeping everything carries its own legal risk

The best case for the haystack is that storage feels cheap. But neither text tells you what a useful record contains, and other law pulls in the opposite direction. Data protection law requires that personal data be kept “for no longer than is necessary” for its purpose (Regulation (EU) 2016/679, art. 5(1)(e)), and AI transcripts are often full of personal data. An organization that answers the evidence question by keeping everything forever has swapped one compliance problem for another.

There is a second honest complication. The AI Act duties quoted above apply to high-risk systems, and the UK code is voluntary. Many small firms are not legally required to do any of this. The customer still wants an answer.

So the useful question is narrower. For one consequential AI-assisted decision, what is the smallest set of facts that would let a stranger rebuild what happened and judge whether the controls operated?

The receipt: five things a record must answer

Operational AI Readiness answers that with an evidence record built around the decision rather than the infrastructure (Bakos, 2026). It needs to tell you:

  • which system identity and version produced the output, meaning the model and also the prompt template and retrieval configuration in force at the time;
  • what the system had in front of it, meaning the sources or documents it drew on;
  • what it produced;
  • what a human did next, including any approval, override or escalation and the reason given;
  • and what was finally executed, and when.

Those fields belong in one linked trace, not scattered across five tools owned by four teams. Appendix B of the book sets out each field and the reason it exists, and the retention period is treated as a deliberate, documented decision rather than whatever the default happened to be. That is FairByDesign’s design, not a legal standard, but it maps directly onto the purposes the law names.

The verdict: smaller, sharper, explainable

The quiet benefit of designing evidence this way is that it shrinks. A decision-level record can often omit large volumes of raw text, or keep it for a shorter time, because the fields that carry the explanation are small. You keep what explains, for as long as you might need to explain it, and you can say why.

What a record still can’t tell you

But here the story turns. Suppose you build all of this. You have an inventory, named owners, an oversight design and a clean evidence record for every consequential decision. You still have not proved that your controls work. Records show what happened. Whether a control actually operated, and would operate again, is a different and harder question, and almost nobody is testing it. That is next.

Go further. Appendix B of Operational AI Readiness is a complete evidence logging schema with the logic behind every field, and Chapter 6 explains how to build it into a live workflow. The complete Standard edition is free: read or download it. To see how your own workflow scores on the evidence spine, try the free readiness assessment.

References

Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live. FairByDesign. https://fairbydesign.org/operational-ai-readiness/

Department for Science, Innovation and Technology. (2025, January 31). Code of practice for the cyber security of AI. GOV.UK. https://www.gov.uk/government/publications/ai-cyber-security-code-of-practice/code-of-practice-for-the-cyber-security-of-ai

Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). (2016). Official Journal of the European Union, L 119, 1–88. http://data.europa.eu/eli/reg/2016/679/oj

Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (2024). Official Journal of the European Union, L, 2024/1689. http://data.europa.eu/eli/reg/2024/1689/oj


FairByDesign is reader-funded. Give once ($5, $10, $25, $50 or any amount), give monthly ($10 or $25), or become a patron.

Give $5 · Give $10 · Give $25 · Give $50 · Any amount · $10 a month · $25 a month · Become a patron

Support never buys editorial control.

Leave a Reply