The Case Against Independent AI Scrutiny, Taken Seriously

FairByDesign exists on the premise that independent scrutiny of AI matters. Here is the strongest case we can build against that premise, and what survives it.

FairByDesign Deepen the Debate title card: “The Case Against Independent AI Scrutiny”

We have spent this month arguing that AI governance is a power question and that someone independent has to check the claims. It would be convenient to stop there. But a publication that tells you to scrutinise everyone else’s incentives and never examines its own is doing marketing, not journalism. So here is the strongest case we can assemble against the idea that independent outsiders, including us, should be scrutinising AI systems at all. We have tried to make it the version its best advocates would recognise.

If you have ever thought “who are these people to judge?”, this one is for you.

Argument one: access is itself a risk

The most serious objection is about security. Toby Shevlane argued that rather than releasing powerful AI systems openly, developers should offer “controlled, arm’s length interactions,” usually through an interface, so that dangerous capabilities do not spread and systems cannot be modified or reverse-engineered (Shevlane, 2022). Every extra party with deep access to a frontier model is another place weights can leak, another set of credentials to steal and another route for a capability to escape into the world. On this view, independent scrutiny is not free. The more meaningful the access, the larger the attack surface. Even researchers who want more outside access concede the point. Bucknall and Trager recommend “minimal sufficient” access for researchers rather than everything they ask for (Bucknall & Trager, 2023). Casper and colleagues, arguing that black-box access is insufficient for rigorous audits, spend real effort on the technical, physical and legal safeguards that deeper access would require (Casper et al., 2024).

Argument two: outsiders are often not competent enough

The second objection is about competence. Frontier systems are complex, and the people best placed to understand them work inside the companies that build them. An outside reviewer with a fraction of the information, compute and context may produce findings that are wrong, misleading or out of date by the time they are published. The audit field itself is uneven. Birhane and colleagues, surveying audits by regulators, law firms, civil society, journalists, academics and consultancies, describe the ecosystem as “muddled and imprecise,” and find that only some audits lead to any accountability outcome at all (Birhane et al., 2024). An outsider who confidently publishes a flawed test can do more damage to public understanding than silence would.

Argument three: scrutiny becomes ritual

The third objection is older than AI. Michael Power’s study of the “audit society” found that verification often turns into ritual. It produces comfort rather than insight, demands more of it than it can deliver, and reshapes organisations around what is easy to check rather than what matters (Power, 1997). Goodman and Tréhu bring the warning to AI. Poorly designed, unstandardised or non-independent audits can be “at best meaningless and at worst even excuses harms” (Goodman & Tréhu, 2022). On this view, adding more outside reviewers does not fix the problem. It adds more stamps, and every stamp lets someone say “it was independently reviewed.”

Argument four: independent of whom?

The fourth objection is aimed squarely at people like us. “Independent” is a claim, not a property. Independent media have funders, audiences, career incentives and a commercial interest in alarming stories. A publication funded by readers who already believe AI needs more oversight has every incentive to keep finding that it does. Researchers who study algorithmic auditing have noted that the field lacks accreditation, and that results and methods are often not disclosed (Costanza-Chock et al., 2022). If professional auditors have that problem, a small independent publication has it more. Why should anyone trust our independence more than a company’s internal safety team, which at least sees the system?

Argument five: the public sector should do it, not freelancers

The last objection is institutional. If scrutiny matters, it should be done by bodies with legal mandates, security clearances, accountability to elected officials and the power to compel information, such as national AI safety institutes and regulators. A patchwork of independent outsiders muddies the picture, duplicates effort and lets governments off the hook for building real capacity.

What survives

This is FairByDesign’s reply, and it is labelled as ours. Some of these arguments land. The security argument is right that access has costs, and it is why our own first test, Evidence Lab 001, uses a fully simulated environment, synthetic data and a published fake signing key, and never touches a real system or credential. The competence argument is right that outsiders can be wrong, and it is why we publish our method before our results and invite people to attack it. The ritual argument is right that a stamp can replace a look, and it is why we refuse to issue stamps. Our job is to show the evidence, not to certify anything. The independence argument is right that independence is a claim, and it is why our funding policy is public, why supporters cannot buy editorial control, and why we are publishing this piece.

What does not survive, in our view, is the conclusion that outside scrutiny should therefore be minimal. Every one of these arguments is a reason to do scrutiny carefully. None of them shows that the people who build and sell a system should be the only ones who tell us whether it works. The institutional argument is the strongest, and we agree that public bodies should do most of this work. But even well-resourced public institutions publish incident reports that outsiders cannot yet check against the underlying evidence, and the public record then depends on what they choose to release. Independent scrutiny is not a replacement for institutions. It makes it harder for any single institution to be the only voice on whether its own systems work.

We hold that position provisionally. If you think we are wrong, we would like to hear the version of this argument we have not managed to make.

Your turn

This is an open invitation. If you work in AI security, at a lab, at a safety institute, in auditing, in standards or anywhere else, and you think independent scrutiny is overrated, dangerous or done badly, write to us at info@fairbydesign.org. Tell us which argument we underplayed and what we should change. With your permission, we will publish the strongest responses, including the ones that disagree with us, and answer them on the podcast. Published responses can run to 800 words, should be sent to info@fairbydesign.org, and may be edited for length and clarity, always with the author’s approval of the final text. Disagreement is welcome here. Fake consensus is the thing we are trying to avoid.

Why this needs independent support: a publication that argues against itself in public can only do it if nobody with a stake in the answer is paying for it. Give once ($5, $10, $25, $50 or any amount), give monthly ($10 or $25), or become a patron → fairbydesign.org/patrons-2/

References

Birhane, A., Steed, R., Ojewale, V., Vecchione, B., & Raji, I. D. (2024). AI auditing: The broken bus on the road to AI accountability. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) (pp. 612–643). IEEE. https://doi.org/10.1109/SaTML59370.2024.00037

Bucknall, B. S., & Trager, R. F. (2023). Structured access for third-party research on frontier AI models: Investigating researchers’ model access requirements. Oxford Martin AI Governance Initiative. https://www.oxfordmartin.ox.ac.uk/publications/structured-access-for-third-party-research-on-frontier-ai-models-investigating-researchers-model-access-requirements

Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., Sharkey, L., Krishna, S., Von Hagen, M., Alberti, S., Chan, A., Sun, Q., Gerovitch, M., Bau, D., Tegmark, M., … Hadfield-Menell, D. (2024). Black-box access is insufficient for rigorous AI audits. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (pp. 2254–2272). ACM. https://doi.org/10.1145/3630106.3659037

Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022). Who audits the auditors? Recommendations from a field scan of the algorithmic auditing ecosystem. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 1571–1583). ACM. https://doi.org/10.1145/3531146.3533213

Goodman, E. P., & Tréhu, J. (2022). AI audit-washing and accountability. German Marshall Fund of the United States. https://www.gmfus.org/news/ai-audit-washing-and-accountability

Power, M. (1997). The audit society: Rituals of verification. Oxford University Press.

Shevlane, T. (2022). Structured access: An emerging paradigm for safe AI deployment. In J. B. Bullock, Y.-C. Chen, J. Himmelreich, V. M. Hudson, A. Korinek, M. M. Young, & B. Zhang (Eds.), The Oxford handbook of AI governance (pp. 604–618). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780197579329.013.39

Leave a Reply