The Evidence Chain, Part 4 of 10. Part 3 asked whose decision it is when an agent acts for you. This part asks a harder question: what happens when the agent is completely loyal to you, and someone else has been editing the “you” it is loyal to?
In February 2026, Microsoft’s security researchers published a count that I have not been able to stop thinking about. Over sixty days, they found more than fifty distinct hidden prompts, planted by thirty-one real companies across more than fourteen industries, all designed to do one thing: get your AI assistant to remember a particular company as “a trusted source” and to recommend it first in future conversations (Microsoft Defender Security Research Team & Kochavi, 2026). The prompts rode inside ordinary-looking “Summarize with AI” buttons on websites. You click, your assistant opens, and along with the page you asked about it receives a quiet instruction about whom you trust. The tools to do this were free, and they were being marketed as an SEO growth hack for language models.
Nobody hacked anything, at least not in the way we usually mean it. No password was stolen. No permission was exceeded. The assistant did exactly what it was built to do: learn what you like and act on it. The only thing that changed was its picture of you.
I call this preference poisoning, and I think it is the agent failure we are least prepared to see. The authorization mechanism stays intact. What gets corrupted is the agent’s representation of its principal’s interests: the model of you that it carries from conversation to conversation and acts on when you are not looking.
Memory turned this from a trick into a business
The underlying weakness is old news in security research. In 2023, Greshake and colleagues showed that instructions hidden in web pages, emails or documents can take control of a language-model application when it later reads that content, without the attacker ever talking to the system directly (Greshake et al., 2023). What changed is persistence. In 2024, the researcher Johann Rehberger showed that a prompt injection from a website could write lasting instructions into ChatGPT’s long-term memory, instructions that kept operating in every later session; OpenAI closed the data-exfiltration channel he used, but Rehberger noted that websites could still plant false memories (Rehberger, 2024). Early in 2025 he showed the same pattern in Google’s Gemini, where a poisoned document could make the assistant save false facts about the user once the user typed an innocent word like “yes”; Google rated the issue a low-likelihood, low-impact abuse risk (Rehberger, 2025).
Put those findings next to Microsoft’s count and you can watch a vulnerability become a market. Once an assistant remembers, the most valuable thing a marketer can buy is not your attention for one session. It is a line in the file that describes who you are. The OWASP GenAI Security Project now lists “Memory & Context Poisoning” as ASI06 in its top ten risks for agentic applications, and the entry’s lead put the point plainly in May 2026: memory should be treated as part of the attack surface (Habler, 2026; OWASP GenAI Security Project, 2025).
The quiet pressure on what your agent believes
Memory is only one channel. The other is what the agent reads when it goes looking. Researchers have shown that website owners can raise their visibility in generative-engine answers by up to 40 percent through content optimization (Aggarwal et al., 2024), that a short “strategic text sequence” added to a product page can push that product up a model’s recommendations (Kumar & Lakkaraju, 2024), and that such preference manipulation works against deployed systems including Bing and Perplexity, setting up a prisoner’s dilemma in which every seller is pushed to manipulate and answer quality degrades for everyone (Nestaas et al., 2024).
Then there is the operator. When OpenAI launched in-chat purchasing in September 2025, it said product results were organic and unsponsored and that merchants pay a fee on completed sales (Bellan, 2025). In January 2026 it began testing advertising in ChatGPT, with the company’s applications chief saying that ads will not influence the answers ChatGPT gives (Rijo, 2026). I take those commitments seriously, and I also notice that, from where you sit, there is no way to check them. The UK’s Competition and Markets Authority made the same observation in March 2026: agents may steer people toward products that are more profitable but less suited to their needs, and an agent that remembers your preferences can make that steering less visible (Competition and Markets Authority [CMA], 2026a).
Why every control waves it through
Almost every safeguard we build for agents asks one of two questions: was this action permitted, and did a human approve it? Preference poisoning passes both. The purchase sits within the spending limit. You may even have clicked approve, because the agent presented the option as the obvious choice given “your preferences.” Google’s Agent Payments Protocol, launched in September 2025, is a good example of serious engineering aimed at the first question: it creates cryptographically signed mandates that leave a non-repudiable trail from your intent to the cart to the payment (Parikh & Surapaneni, 2025). That trail proves you authorized the purchase. It cannot prove that the preference behind your authorization was yours.
This is why I think the question that matters is not whether the agent was authorized, but whether it can show that its decision stayed faithful to your actual preferences rather than to a contaminated record of them. Our own Evidence Lab 001 protocol tests a small, adjacent version of this: whether a legitimate action stays legitimate after the agent has read an unrelated poisoned document.
Who gets to write the file about you
Strip away the engineering and this is a question about power. Who decides what goes into the description of you that your agent acts on? Today, in practice, it is a mix of you, the operator that sets the memory policy, and anyone whose content the agent happens to read. Who can intervene? Mostly the operator, and only if it notices. Who carries the consequences? You do, in the form of the insurer you did not choose, the subscription you did not need, the source you were nudged to trust.
That asymmetry falls hardest on the people who use these systems most intimately. Common Sense Media reported in July 2025 that nearly three in four US teenagers had used AI companions (Common Sense Media, 2025), and two months later the US Federal Trade Commission opened an inquiry into companion chatbots that asks, among other things, how the companies monetize user engagement (Federal Trade Commission [FTC], 2025). Young people are not naive about technology; many understand these systems better than the adults regulating them. But nobody, of any age, can audit a memory they cannot see, and the generation growing up with remembering assistants deserves to know who else is writing in the file.
The prediction
My prediction is that a significant share of future disputes over AI agents will turn on corrupted preferences rather than broken permissions, and that the evidence needed to resolve them, namely where a preference came from, when it was recorded and what changed it, mostly does not exist in today’s systems. The agent will be able to prove it was authorized. It will not be able to prove that the person it thought it was serving was really you. That is a solvable problem, but only if someone asks for the evidence before the disputes begin.
For Mercury Intelligence members: the section below walks the Microsoft case through the evidence chain, maps it clause by clause against the EU AI Act, UK consumer guidance and US enforcement, sets out how to give preferences a provenance and audit for drift, and tests the argument against its strongest critics.
Support independent investigation. This story exists because someone read a security blog closely and asked who benefits when an assistant’s memory is for sale. No advertiser or platform pays for that question to be asked, and that is the point. Give once ($5, $10, $25, $50 or any amount), give monthly ($10 or $25), or become a patron: fairbydesign.org/patrons-2.
References
Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., & Deshpande, A. (2024). GEO: Generative engine optimization. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM. https://arxiv.org/abs/2311.09735
Bellan, R. (2025, September 29). OpenAI takes on Google, Amazon with new agentic shopping system. TechCrunch. https://techcrunch.com/2025/09/29/openai-takes-on-google-amazon-with-new-agentic-shopping-system
Common Sense Media. (2025, July 16). Talk, trust, and trade-offs: How and why teens use AI companions. https://www.commonsensemedia.org/research/talk-trust-and-trade-offs-how-and-why-teens-use-ai-companions
Competition and Markets Authority. (2026a, March 9). Agentic AI and consumers. GOV.UK. https://www.gov.uk/government/publications/agentic-ai-and-consumers/agentic-ai-and-consumers
Competition and Markets Authority. (2026b, March 9). Using AI agents: Complying with consumer law. GOV.UK. https://www.gov.uk/government/publications/complying-with-consumer-law-when-using-ai-agents
European Commission. (2025, February 4). Commission publishes the guidelines on prohibited artificial intelligence (AI) practices, as defined by the AI Act. https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act
Federal Trade Commission. (2025, September 11). FTC launches inquiry into AI chatbots acting as companions [Press release]. https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-launches-inquiry-ai-chatbots-acting-companions
Federal Trade Commission. (2026, July 1). FTC seeks public comment on policy statement addressing AI accuracy [Press release]. https://www.ftc.gov/news-events/news/press-releases/2026/07/ftc-seeks-public-comment-policy-statement-addressing-ai-accuracy
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (pp. 79–90). ACM. https://doi.org/10.1145/3605764.3623985
Habler, I. (2026, May 13). Memory is a feature. It is also an attack surface. OWASP GenAI Security Project. https://genai.owasp.org/blog/memory-is-a-feature-it-is-also-an-attack-surface/
Kumar, A., & Lakkaraju, H. (2024). Manipulating large language models to increase product visibility (arXiv:2404.07981). arXiv. https://arxiv.org/abs/2404.07981
Microsoft Defender Security Research Team, & Kochavi, N. (2026, February 10). Manipulating AI memory for profit: The rise of AI recommendation poisoning. Microsoft Security Blog. https://www.microsoft.com/en-us/security/blog/2026/02/10/ai-recommendation-poisoning/
Nestaas, F., Debenedetti, E., & Tramèr, F. (2024). Adversarial search engine optimization for large language models (arXiv:2406.18382). arXiv. https://arxiv.org/abs/2406.18382
OWASP GenAI Security Project. (2025, December 9). OWASP top 10 for agentic applications for 2026. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
Parikh, S., & Surapaneni, R. (2025, September 16). Powering AI commerce with the new Agent Payments Protocol (AP2). Google Cloud Blog. https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol
Privacy Laws & Business. (2026, May 15). EU proposal on Digital Fairness Act expected by the end of 2026. https://www.privacylaws.com/news/eu-proposal-on-digital-fairness-act-expected-by-the-end-of-2026/
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). (2024). Official Journal of the European Union, L, 2024/1689. http://data.europa.eu/eli/reg/2024/1689/oj
Rehberger, J. (2024, September 20). Spyware injection into your ChatGPT’s long-term memory (SpAIware). Embrace The Red. https://embracethered.com/blog/posts/2024/chatgpt-macos-app-persistent-data-exfiltration/
Rehberger, J. (2025, February 10). Hacking Gemini’s memory with prompt injection and delayed tool invocation. Embrace The Red. https://embracethered.com/blog/posts/2025/gemini-memory-persistence-prompt-injection/
Rijo, L. (2026, January 16).
OpenAI finally pulls trigger on ChatGPT ads after monthslong delay.
PPC Land. https://ppc.land/openai-finally-pulls-trigger-on-chatgpt-ads-after-monthslong-delay/Change control for prompts, retrieval and tools: Chapter 8 of Operational AI Readiness (free).
Leave a Reply
You must be logged in to post a comment.