The Control File
AI Cybersecurity, Explained. Plain-language security for everyone, with practical steps for the organizations that answer for AI at work.
Picture the person who pays the bills at a small firm. It is Friday afternoon. An email arrives from a supplier she has worked with for years: same tone, right invoice number, a polite note that their bank details have changed. No spelling mistakes. No odd greeting. Nothing that feels wrong. (This scene is illustrative, but it is the shape of a crime that cost reported victims more than $3 billion in 2025.)
For years we told her to look for the typos. That advice is running out of road. In December 2024 the FBI warned that criminals use generative AI to commit fraud faster and at greater scale, and that these tools can correct the human errors that “might otherwise serve as warning signs” (FBI, 2024).
Here is my proposition: AI has broken the typo test, but it has not broken the defences that actually work, because those never depended on how well a scam was written. Let’s test it.
The scam got a ghostwriter
Phishing is a message designed to trick you into clicking a link, opening a file, handing over a password or sending money. It is already the most common crime reported to the FBI. In 2025, the Bureau’s Internet Crime Complaint Center received 191,561 phishing and spoofing complaints, more than any other category (FBI IC3, 2026, p. 7).
It is also often the first step in business email compromise, where criminals pose as a boss or supplier to redirect payments. That is the Friday-afternoon email above, and it cost reported victims more than $3 billion in 2025 (FBI IC3, 2026, pp. 7, 9). Who pays? The small firm that loses the money, and the bookkeeper who gets blamed.
AI makes three things easier for the people on the other end. First, fluency: messages in good English, or any language, with no giveaway errors. Second, research: gathering public details about a target to make a message personal. Third, scale: doing both for thousands of people at almost no extra cost.
The experiment: 101 people, four kinds of email
A study by Fred Heiding, Bruce Schneier and colleagues tested this with real people. They sent 101 participants different kinds of phishing emails. The results:
- Generic phishing emails: a 12% click rate.
- Emails written by human experts: 54%.
- Emails fully generated by an AI system, including the research on each target: also 54%.
- AI emails with a human in the loop: 56%.
The AI’s research produced accurate and useful profiles in 88% of cases, and the authors estimate automation could make large phishing campaigns up to 50 times more profitable (Heiding et al., 2024). It is one study with a modest sample, but the direction is clear. Personalised phishing used to take skill and time, and now it takes neither.
The UK’s National Cyber Security Centre (NCSC) reached the same view in its 2024 assessment. It judged that AI “will almost certainly increase the volume and heighten the impact of cyber attacks,” and that it makes social engineering “more effective, efficient, and harder to detect,” with the biggest boost for less-skilled criminals (NCSC, 2024).
The complication: this is not a brand-new monster
Here is the best argument against the panic, and it comes from the same assessment. The NCSC described all this as an evolution of existing tactics, not a new kind of attack (NCSC, 2024). The trick is the one it always was: get you to act before you check.
So if the message itself can be perfect, protection has to come from things the message cannot fake.
Key idea: phishing-resistant authentication
Phishing-resistant authentication is a way of logging in that cannot be handed to a fake website, even by someone who has been completely fooled. Passkeys and physical security keys do this by checking the real website’s address automatically before they work. It matters because it protects you on the one day the scam is good enough, which, with AI-written phishing, may be any day.
What a perfect email still cannot do
It cannot steal a login that refuses to be handed over. The US Cybersecurity and Infrastructure Security Agency (CISA) points out that not all multi-factor authentication is equally secure: codes sent by text or typed from an app can be tricked out of people. It calls phishing-resistant methods, such as FIDO security keys and passkeys or smart-card systems, “the gold standard,” while stressing that “any form of MFA is better than no MFA” (CISA, 2022, pp. 1–2). If a service you use offers passkeys, turning them on is one of the most effective security steps you can take.
It cannot answer a phone it does not own. The FBI’s consistent advice is to verify unexpected requests by contacting the person or organization on a number or address you already know, not one in the message (FBI, 2024). An AI can write a perfect email from “your bank,” but it cannot make your bank’s real phone number reach the criminal. Our bookkeeper calls the supplier on the number in her files, and the scam ends.
It cannot talk its way past a rule. For organizations, payment changes and large transfers should need an independent callback and a second approver, whatever the email says. A well-written message then hits a process, not just a person.
It cannot be sure the other side has no AI. The same tools cut both ways. In the Heiding study, the researchers also asked AI models to judge whether emails were suspicious, and the best-performing model detected the intent of phishing emails more than 90% of the time with few false alarms (Heiding et al., 2024). The NCSC likewise notes that AI can help defenders detect and triage attacks (NCSC, 2024). Email filtering that uses these techniques is worth asking your provider about, as long as it is treated as one layer, not a guarantee.
Training still matters, but its focus should shift. Instead of “spot the typo,” teach people to notice the situation: an unexpected request, urgency or secrecy, a new link to log in, a change of payment details. Those are the features that criminals cannot remove, because they are the point of the scam.
The verdict
The proposition holds. The evidence says AI-written phishing is as convincing as the work of skilled humans and far cheaper to produce, so any defence that relies on bad writing is finished. But the defences that held up best, passkeys, callbacks on a known number and payment rules with a second approver, were never about spotting bad writing. The scam got better at talking. These defences don’t listen.
What we still don’t know
We cannot say how much phishing today is AI-written. The FBI’s 2025 report counted only 803 phishing complaints that referenced AI (FBI IC3, 2026, p. 41), but that reflects what victims noticed and reported, not what attackers used, and a fluent AI-written email is by design hard to recognize. The FBI also notes that not all business email compromise tactics are AI-enabled (FBI IC3, 2026, p. 39). The Heiding study involved 101 volunteers in a research setting, so its click rates may not match real-world campaigns. And the NCSC assessment was written in early 2024 as a forecast to 2025; it is a judgement, not a measurement.
Which leaves the question worth watching: if almost nobody can tell an AI-written scam from a human one, will the official numbers ever show us how big this really is?
What to do this week
- For yourself: turn on passkeys or another form of multi-factor authentication for email, banking and any account that can reset others.
- When a message asks you to act: pause on urgency, secrecy or a new link to log in, and check by contacting the sender a way you already know.
- If you run an organization: prioritise phishing-resistant authentication for administrators and finance staff, and make payment changes impossible without a callback.
- If you run training: retire “look for spelling mistakes” and teach the warning signs of the situation instead.
Go further. Operational AI Readiness is FairByDesign’s free field manual for AI that is already live. Chapter 7, Security Controls for LLM Apps and Agent Workflows, covers the security boundaries that matter as AI reads and writes more of your organization’s email, and Chapter 9 shows how to build approval gates that a convincing message cannot talk its way past. Read it free.
Your next step: scam techniques change month to month. Get the free Mercury Brief for plain-language updates on what is new and what still works.
FairByDesign is reader-funded. If this helped, get the free Mercury Brief or see ways to support. Support never buys editorial control.
References
Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live. FairByDesign. https://fairbydesign.org/operational-ai-readiness/
Cybersecurity and Infrastructure Security Agency. (2022, October). Implementing phishing-resistant MFA [Fact sheet]. https://www.cisa.gov/sites/default/files/publications/fact-sheet-implementing-phishing-resistant-mfa-508c.pdf
Federal Bureau of Investigation. (2024, December 3). Criminals use generative artificial intelligence to facilitate financial fraud (Alert No. I-120324-PSA). Internet Crime Complaint Center. https://www.ic3.gov/PSA/2024/PSA241203
Federal Bureau of Investigation Internet Crime Complaint Center. (2026). 2025 IC3 annual report. https://www.ic3.gov/AnnualReport/Reports/2025_IC3Report.pdf
Heiding, F., Lermen, S., Kao, A., Schneier, B., & Vishwanath, A. (2024). Evaluating large language models’ capability to launch fully automated spear phishing campaigns: Validated on human subjects (arXiv:2412.00586) [Preprint]. arXiv. https://arxiv.org/abs/2412.00586
National Cyber Security Centre. (2024, January 24). The near-term impact of AI on the cyber threat [Assessment]. https://www.ncsc.gov.uk/report/impact-of-ai-on-cyber-threat



Leave a Reply
You must be logged in to post a comment.