The Control File
From the Field Manual. Each week we take one chapter of our free book Operational AI Readiness and turn it into something you can use. This week: Chapter 8, on keeping track of what changed.
Picture a shop owner on a Monday morning. A customer is on the phone, annoyed: the website’s AI assistant gave her an answer last week that it never used to give. The owner calls the vendor. The vendor checks and says, cheerfully, “Nothing’s changed. You’re still on the same model.” (An illustrative scene, not a real case.)
That answer is usually true. It is also, very often, the wrong answer to the question.
Here is my proposition: “same model” is one of the least reassuring sentences in AI. What you experience is shaped by everything wrapped around the model, and that wrapping can change overnight without anyone writing it down.
Exhibit A: three changes, one model
The public record of one widely used chatbot shows how. In February 2025, according to TechCrunch, an xAI engineering lead said an employee had instructed the company’s Grok chatbot “to ignore sources that mentioned Musk or Trump spreading misinformation”; the change was reverted after users noticed (Wiggers, 2025).
In May 2025, xAI said that at about 3:15 a.m. on May 14 someone had made an “unauthorized modification” to the prompt behind Grok’s bot on X, directing it toward a “specific response” on a “political topic.” The company promised to publish Grok’s system prompts on GitHub with a changelog, add “additional checks and measures” so employees could not change the prompt without review, and set up a 24/7 monitoring team (Wiggers, 2025).
Then, in July 2025, Grok posted antisemitic content on X. In its apology, xAI said the root cause was “an update to a code path upstream of the @grok bot,” active for 16 hours, which made the bot “susceptible to existing X user posts.” The company stressed that this was “independent of the underlying language model” (Alsharif, 2025).
Three serious behavior changes in six months. By xAI’s own account, each came from instructions or code around the model, not from a new model. Notice who found the first one: users. Not a review, not a test. The people on the receiving end.
Who benefits from “nothing changed,” and who pays
A narrow definition of change suits the people shipping it: prompt edits are quick and need no paperwork. The cost lands on the customer given a different answer and the business that cannot explain why.
Chapter 8 of Operational AI Readiness opens with exactly this point: “model change” is too small a phrase for what organizations need to govern. The changes that matter most are often prompt edits, new retrieval sources, added connectors, policy updates, tool permissions, vendor-side safety adjustments, and interface changes that alter how reviewers see the output (Bakos, 2026, p. 58). Each can feel incremental. Together they decide how the system behaves. As Viktoria Bakos writes, a workflow that cannot remember its own meaningful changes “will eventually be governed by institutional amnesia” (Bakos, 2026, p. 58).
Key idea: change control for AI workflows
Change control means keeping a short, reliable record of every meaningful change to a live AI system (to its instructions, data sources, tools, interface or vendor model), testing those changes in proportion to their risk, and deciding in advance what would trigger a rollback. It matters to you because an AI tool that answered your question well last month may answer differently today, and without a change record nobody can tell you why.
The best argument against me: this is just more red tape
It’s a fair objection. Make every comma a change ticket and small teams will route around the process, or stop improving the system at all.
The chapter agrees. Change control should be “proportional, not theatrical” (Bakos, 2026, p. 59). Fixing a typo in a help message is not the same as widening what a system can read or do. The skill is telling them apart.
It offers a quick test for any change (Bakos, 2026, p. 59). Could it alter the substance, confidence or tone of outputs in consequential cases? Could it change what sources, tools or actions the system can reach? Could it affect a reviewer’s ability to interpret, challenge or override the output? Could it widen foreseeable misuse or shrink your ability to contain problems? And if it caused harm, would you wish you had formally recorded and tested it? If the answers lean toward yes, the change deserves more than a casual merge note.
The changes nobody thinks to count
Three further principles stand out. Validation should match the risk the change introduces, because “changed behavior should be observed before it is trusted” (Bakos, 2026, p. 60).
Changes to what reviewers see count too. Hiding source excerpts to make a screen look cleaner can weaken oversight without touching the model, and the book calls that “a control change” (Bakos, 2026, p. 60).
And vendor-side changes count, even when you did not make them (Bakos, 2026, p. 61). If you build on someone else’s AI service, their updates are your changes.
The verdict
On this record, “we’re still on the same model” tells you almost nothing about whether your system behaves the same way. The fixes xAI promised in May 2025 were reviews, a public changelog and monitoring: not a new model. A light, honest change record is not bureaucracy. It is the difference between answering a customer’s question and shrugging at it.
What your team can do this month
- Widen the definition of “release.” Agree in writing that prompt edits, new data sources, new connectors or tool permissions, reviewer-screen changes and vendor notices all count as changes to the AI system.
- Keep a light change record. The chapter’s list is short: what changed, why, what risks it may affect, what validation ran, who approved it, when it went live, and what rollback or extra monitoring applies (Bakos, 2026, p. 62).
- Put the prompt under review. If one person can edit a production prompt without a second pair of eyes, fix that first. xAI’s own remedy in May 2025 was to stop unreviewed prompt changes (Wiggers, 2025).
- Write rollback triggers now, “before deployment, not after regret” (Bakos, 2026, p. 61): rising override rates, more complaints in one category, unexplained shifts in output.
- Keep a small regression set of consequential questions and run it after every meaningful change, including vendor updates.
If you’re on the receiving end
AI products change constantly, often silently. If a service you rely on starts behaving differently, more confident, more evasive, or suddenly wrong about something it used to get right, you are probably not imagining it. It is reasonable to ask the provider what changed and when. An organization with real change control can answer. One without it will say the model is the same.
What we still don’t know
Our account of the three Grok incidents relies on xAI’s own statements as reported by TechCrunch and NBC News. We have not independently verified who made the changes, how review was bypassed, or whether the prompts xAI publishes are complete; some commentators have questioned that. We are not suggesting other companies have had the same incidents. We use this case because its public statements show, unusually clearly, that behavior can change while the model stays the same.
Which leaves the bigger question: for the AI tools that quietly decide things about you, who is keeping the changelog, and would they show it to you if you asked?
Questions you can ask
- If you are affected: “Has the AI system that handled my case changed since my last contact, and is there a record of what changed?”
- If you are affected: “If an answer I was given turns out to be wrong, can you tell which version of the system gave it?”
- If you run AI at work: “Could we list every change to this workflow’s prompts, sources and tools in the last 30 days, and who approved each one?”
- If you run AI at work: “What would make us roll back, and who decides?”
Go further. Operational AI Readiness is FairByDesign’s free field manual for AI that is already live. Chapter 8, “Change Management for Models, Prompts, Tools, and Retrieval,” covers proportional validation, reviewer-facing changes, rollback conditions and vendor updates, with worked examples. Read the chapter free, or check your change control with the free readiness assessment.
Your next step: get the free Mercury Brief. It brings you the next chapter in this series and the public-record cases behind it, so you hear about what changed before it surprises you.
FairByDesign is reader-funded. If this helped, get the free Mercury Brief or see ways to support. Support never buys editorial control.
References
Alsharif, M. (2025, July 12). AI chatbot Grok issues apology for antisemitic posts. NBC News. https://www.nbcnews.com/news/us-news/ai-chatbot-grok-issues-apology-antisemitic-posts-rcna218471
Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live (Free ed.). FairByDesign. https://fairbydesign.org/operational-ai-readiness/
Wiggers, K. (2025, May 15). xAI blames Grok’s obsession with white genocide on an “unauthorized modification.” TechCrunch. https://techcrunch.com/2025/05/15/xai-blames-groks-obsession-with-white-genocide-on-an-unauthorized-modification
xAI. (n.d.). grok-prompts [Source code repository]. GitHub. Retrieved October 15, 2026, from https://github.com/xai-org/grok-prompts




Leave a Reply
You must be logged in to post a comment.