Securing AI-Generated Code: Who Checked the Part That Guards Your Password?

AI tools now write part of the software that holds your logins and payments. The UK’s cyber agency says that is fine for some code and risky for the rest. The real question is whether anyone checks the parts that…

FairByDesign The Control File title card: “Securing AI-Generated Code: Who Checked the Part That Guards Your Password?”

The Control File

AI Cybersecurity, Explained. Plain-language security for everyone, with practical steps for the organizations that answer for AI at work.

You run a small clinic, or a shop that takes online orders. Last spring someone on your team rebuilt the booking site over a weekend with an AI coding tool. Customers log in, change their details, pay. Nobody has looked closely at the part that checks passwords. If that part fails, the letter to your customers goes out with your name on it, not the AI tool’s.

Here is our proposition: AI-written code is not the danger. Unchecked AI-written code in the few places that guard people’s logins, money and records is. Tool makers selling speed collect the benefit. Customers whose passwords sit behind code nobody read, and the owner who answers for the breach, pay.

This matters to you even if you will never write a line of code. The apps on your phone, the portal where you see your test results, the form you fill in to claim a benefit: more and more of that software is now partly written by AI. The question is not whether that is happening. It is whether anyone is checking the parts that protect you.

The cyber chief who didn’t say stop

In March 2026, the head of the UK’s National Cyber Security Centre stood on stage at the RSAC Conference in San Francisco and talked about something most people outside software had never heard of: vibe coding, the practice of asking an AI tool to write software for you.

Dr Richard Horne did not tell the audience to stop. He called the shift a huge opportunity, “but not without risk of its own,” and said the AI tools used to develop code “must be designed and trained from the outset” so that they do not introduce or spread unintended vulnerabilities (National Cyber Security Centre [NCSC], 2026a).

The same day, the NCSC published a blog arguing that AI-produced code currently poses intolerable risks for many organizations, while also showing “glimpses of a new paradigm” (NCSC, 2026a). In June it followed up with a practical piece that we think deserves a much wider audience than security engineers. Its argument fits in one sentence: different code deserves different levels of care (NCSC, 2026b).

A fast, confident new builder

Think of an AI coding assistant as a very fast new builder who has studied millions of houses, including plenty that were badly built. Ask for a garden shed and you will get a perfectly good shed in minutes.

Ask for the lock on the front door and the builder will produce something that looks like a lock, works when you test it with your own key, and may still have a weakness a burglar would spot. The NCSC puts the root cause plainly: AI models are trained on vast amounts of existing code, “and some of that has security issues” (NCSC, 2026b).

The receipt: ingredients that don’t exist

There is a second, stranger problem. Modern software is assembled from ready-made building blocks called packages, downloaded from public libraries. AI tools sometimes recommend packages that do not exist.

A team of researchers tested 16 AI models across 576,000 generated code samples and found that at least 5.2% of the packages suggested by commercial models, and 21.7% of those suggested by open-source models, were invented (Spracklen et al., 2025).

That sounds like a harmless mistake until you see the trick it enables. An attacker notices which made-up names come up repeatedly, publishes a malicious package under one of them, and waits for the next AI-assisted project to download it. OWASP, the open security community, now lists exactly this scenario in its guidance on AI risks (OWASP GenAI Security Project, 2025).

The analogy is a recipe that calls for an ingredient brand that has never existed. Someone opens a shop with that name and sells you something you should not eat. The only party guaranteed to profit from an AI’s imagination is the attacker paying attention.

Key idea: risk-calibrated oversight

Risk-calibrated oversight means matching the amount of human checking to what is at stake, instead of applying one rule to everything. A demo can be built quickly with little review; the code that guards your password, your money or your medical record needs careful, expert checking before it goes live. For ordinary people, it is the difference between an organization that uses AI everywhere with the same light touch and one that knows which parts of its software can hurt you.

A sliding scale, not a ban

The NCSC describes a spectrum. At one end, a person writes the code and the AI suggests the next line, which they accept or ignore. At the other end, a person gives a broad instruction and lets the AI choose the design, write the code and write its own tests, judging only whether the result seems to work. Most real work sits somewhere in between (NCSC, 2026b).

The NCSC says it is fine to sit toward the fully automated end for prototypes, demonstrations and internal tools where no sensitive data or security functions are involved. It says to move back toward careful human control for login and permission logic, sensitive personal data, anything handling secrets, tokens or credentials, safety-critical systems, and any code where a flaw would have significant consequences (NCSC, 2026b).

The best case against our worry

The strongest argument against us comes from the agency itself. Its framing is refreshingly unpanicked: “The risk isn’t in using AI. The risk is not applying the right safeguards when the stakes are high” (NCSC, 2026b). Most of that weekend booking site is a shed, and sheds are fine.

The research cuts the same way. The package-hallucination rates come from controlled tests on the models available when the study was run, and the researchers also showed that mitigations can reduce the problem substantially (Spracklen et al., 2025). The NCSC even suggests you could use another AI agent to help with the checking (NCSC, 2026b). And the root flaw it names, code learned from code that had security issues, was a human problem long before it was a machine one.

That is a fair point. It just doesn’t rescue the front-door lock. It moves the question from “should we use AI?” to “who decided which parts get checked?”

If you build small things yourself

If you use AI tools to build small things for yourself, a spreadsheet macro, a personal website, a script that renames your photos, the NCSC’s advice suggests you can relax about the low-stakes work. Slow down when what you are building touches other people’s information, takes payments, or asks anyone to log in. In those cases, three habits go a long way:

  • Never paste real passwords, API keys or access tokens into an AI prompt or into code you share. Keep them in a password manager or the secure settings your platform provides.
  • Before installing any package an AI suggests, check that it exists on the official library, has a real history and is widely used. A brand-new package with a plausible name is a warning sign, not a coincidence.
  • Ask a person who knows security to look at anything involving logins or personal data before other people rely on it.

If you run the business: four decisions to write down

Most organizations have not yet written these down. First, sort your code by consequence. Decide, in advance, which parts of your systems count as security-critical: authentication, payments, personal data, secrets handling, anything that can change records or move money.

Second, set the oversight level for each category. For the critical tier, the NCSC lists four verbs: review what the AI produces, understand the code, check it for vulnerabilities, and verify it does what you expect. It adds that you should design the wider system so that a flaw, if exploited, does limited damage (NCSC, 2026b).

Third, treat dependencies as a control point. Keep an approved list of packages and versions, block installs that are not on it, and scan for secrets before code is merged. These are long-established practices; AI simply raises the volume of code that needs them.

Fourth, know where AI-generated code lives. If you cannot say which systems were substantially written by AI tools, you cannot apply a risk-based review to them, and you will struggle to explain a failure afterwards.

The verdict

The evidence supports our proposition. The danger is not the tool. It is the gap between the speed the tool sells and the checking nobody budgeted for, sitting exactly where customers keep their passwords and cards. None of this requires banning AI coding tools. It requires deciding where they are trusted and recording that the decision was followed.

For that booking site, it means one afternoon: find the login and payment code, and have someone who knows security read it.

What we still don’t know

We did not find a reliable public count of security incidents caused specifically by AI-written code, and we have not used figures from secondary sources that we could not trace to a primary study. The NCSC itself notes that its spectrum may shift as models improve, and asks organizations to calibrate “based on today’s reality, not tomorrow’s potential” (NCSC, 2026b).

That is the honest position: the risk is real, measurable in research settings, and still poorly measured in the wild. So here is the question we will keep asking: when the next breach letter lands, will anyone be able to say whether a person or a machine wrote the code that failed?

Questions you can ask this week

  • If you use a service: Does this organization use AI tools to write the code that protects my account, and who reviews that code?
  • If you build with AI: Which parts of what I am building handle logins, money or other people’s data, and have I checked those parts myself or with someone who can?
  • If you lead a team: Have we written down which code is security-critical and what review it needs, and could we show that the review happened on last month’s changes?
  • If you buy software: Will the vendor tell us how AI-generated code is reviewed and how new dependencies are approved?

Go further. Operational AI Readiness is FairByDesign’s field manual for AI that is already live, and the complete Standard edition is free to read and download. Chapter 7, Security Controls for LLM Apps and Agent Workflows, covers the boundaries that keep AI tools from reaching further than they should, and Chapter 8 covers change control for the code, prompts and tools around them. Read it free, or test one workflow against domain 7 (LLM and agent security boundaries) in the free readiness assessment.

Your next step: read Operational AI Readiness free, starting with Chapter 7.


FairByDesign is reader-funded. If this helped, get the free Mercury Brief or see ways to support. Support never buys editorial control.

References

Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live. FairByDesign. https://fairbydesign.org/operational-ai-readiness/

National Cyber Security Centre. (2026a, March 24). NCSC CEO: Seize ‘disruptive’ vibe coding opportunity to make software more secure [News release]. https://www.ncsc.gov.uk/news/ncsc-ceo-seize-disruptive-vibe-coding-opportunity-to-make-software-more-secure

National Cyber Security Centre. (2026b, June 18). The ‘vibe coding spectrum’ approach to AI-assisted software development [Blog post]. https://www.ncsc.gov.uk/blogs/the-vibe-coding-spectrum-approach-to-AI-assisted-software-development

OWASP GenAI Security Project. (2025). LLM09:2025 Misinformation. OWASP Top 10 for LLM Applications 2025. https://genai.owasp.org/llmrisk/llm092025-misinformation/

Spracklen, J., Wijewickrama, R., Sakib, A. H. M. N., Maiti, A., Viswanath, B., & Jadliwala, M. (2025). We have a package for you! A comprehensive analysis of package hallucinations by code generating LLMs (arXiv:2406.10279) [Preprint; to appear in the 2025 USENIX Security Symposium]. arXiv. https://arxiv.org/abs/2406.10279

Leave a Reply