The AI Said It Was Almost Never Wrong. The NIST Framework Asks: Who Checked?

A hospital AI tool was marketed with a severe hallucination rate under one in 100,000. Texas said the figures were likely inaccurate; the company denied it. NIST’s AI risk framework boils down to four questions that put any such claim…

FairByDesign The Control File title card: “The AI Said It Was Almost Never Wrong. The NIST Framework Asks: Who Checked?”

The Control File

It is shift change on a hospital ward. A nurse has a few minutes per patient, and on the screen is a neat AI-written summary of your mother’s condition and treatment. (This scene is illustrative.) The nurse reads it and moves on. Nobody in that moment asks the question that matters most: how often is this thing wrong, and who checked?

That scene is not far from a real case. In September 2024, the Texas Attorney General announced what his office called a first-of-its-kind settlement with Pieces Technologies, a Dallas company selling generative AI to hospitals. At least four major Texas hospitals had been sending patients’ healthcare data to Pieces in real time so that its AI could “summarize” each patient’s condition and treatment for hospital staff (Texas Attorney General, 2024a).

The company’s marketing, according to the state, claimed a “critical hallucination rate” of under 0.001 percent and a “severe hallucination rate” of under one in 100,000 (Texas Attorney General, 2024b, para. 13). That is a remarkable promise. In plain English: almost never wrong.

The claim, and the receipt

The Attorney General’s office said its investigation found those metrics “were likely inaccurate and may have deceived hospitals” about the accuracy and safety of the products (Texas Attorney General, 2024a). Pieces denied any wrongdoing and maintained that it had accurately represented its hallucination rate (Texas Attorney General, 2024b, para. 15).

Look at what the five-year agreement actually requires. If Pieces advertises accuracy figures, it must explain what each one means and how it was calculated. It must give hospitals documentation of the tool’s intended use, its known limitations and risks to patients, and what staff need to “monitor for patterns of inaccuracy” (Texas Attorney General, 2024b, paras. 16–19).

Here is my proposition: an accuracy number without a method is a sales line, not a safety fact, and the most widely cited US framework for AI risk is, at heart, a tool for asking “who checked?” Let’s test that against the framework itself.

The rulebook nobody has to follow

The AI Risk Management Framework, or AI RMF, was published by the US National Institute of Standards and Technology (NIST) on 26 January 2023. It is not a law. NIST describes it as “voluntary, rights-preserving, non-sector-specific, and use-case agnostic”, meant for organizations of any size in any sector (NIST, 2023, p. 2).

It has a companion Playbook of suggested actions, and in July 2024 NIST added a Generative AI Profile applying the framework to chatbots and other generative tools (NIST, n.d.). NIST says the framework is now being revised as part of the White House AI Action Plan, so expect an update. The core ideas below are the ones in version 1.0 (NIST, n.d.).

What “risk” and “trustworthy” really mean

NIST uses a classic definition of risk: a combination of how likely something is to happen and how bad the consequences would be (NIST, 2023, p. 4). A wrong word in a film recommendation and a wrong word in a medication summary are not the same risk.

The framework then lists seven characteristics of trustworthy AI: valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; and fair, with harmful bias managed (NIST, 2023, p. 12). These are not marketing adjectives. Each one is a question you can test: does it work, can it hurt someone, can it be attacked, can we explain it, does it treat people fairly?

Key idea: AI risk management

AI risk management is the ongoing work of asking what could go wrong with an AI system, how likely and how serious it would be, and what you are doing about it, before launch and for as long as the system runs. It matters to you because the risk of a system that summarises, scores or flags people is mostly carried by those people, and risk management is how an organization is pushed to notice problems before you do.

Four questions that would put any brochure on trial

The heart of the framework is four functions. NIST is clear that they are not a checklist or a strict sequence (NIST, 2023, p. 20). Think of them as four questions an organization keeps asking, and watch how each one lands on the Texas case.

Govern: who is responsible, and how do we decide? Govern is about culture, policies and accountability, and NIST describes it as cross-cutting, running through the other three (NIST, 2023, pp. 20–21). Among its outcomes: roles and responsibilities that are documented and clear (GOVERN 2.1), executive leadership taking responsibility for decisions about AI risk (GOVERN 2.3), and policies for the risks that come from third parties such as vendors (GOVERN 6.1) (NIST, 2023, pp. 23–24).

The Texas Attorney General put the hospital side of this plainly: healthcare organizations “must consider whether AI products are appropriate and train their employees accordingly” (Texas Attorney General, 2024a). Someone inside each hospital has to own that decision. Not the vendor. Not “the system”. A person.

Map: what is this system for, and who could it affect? Map establishes context: intended purposes, the setting, the people affected (MAP 1.1), and documentation of the system’s knowledge limits and how its output should be used and overseen by humans (MAP 2.2) (NIST, 2023, pp. 24–26).

A patient summary read by a busy nurse at shift change is a very different context from a draft that a doctor checks line by line. The settlement’s requirement to document intended use, known limitations and known misuses is, in effect, a mapping requirement.

Measure: how do we know how it behaves? This is where the brochure meets the evidence. AI systems, NIST says, “should be tested before their deployment and regularly while in operation” (NIST, 2023, p. 28). The system should be demonstrated to be valid and reliable, with the limits of that evidence documented (MEASURE 2.5), and its behaviour monitored in production (MEASURE 2.4) (NIST, 2023, p. 29).

A figure such as “under one in 100,000” means nothing until you know what counted as a hallucination, which outputs were checked, and by whom. That is exactly what the agreement now requires Pieces to disclose. For a hospital, the Measure question is blunt: did we test it on our own patients and our own workflows, or did we rely on the brochure?

Manage: what do we do about it? Manage means putting resources against the risks you have mapped and measured, and having plans to respond and recover (NIST, 2023, p. 31). Two outcomes stand out. Third-party AI risks are “regularly monitored” with controls applied and documented (MANAGE 3.1). And mechanisms and assigned responsibilities exist “to supersede, disengage, or deactivate” systems whose performance is inconsistent with their intended use (MANAGE 2.4) (NIST, 2023, p. 32).

Translate that to the ward: if staff spot a pattern of wrong summaries, who do they tell, and who can switch the tool off?

The best case against me

Here is the strongest objection, and it is a fair one. A voluntary framework cannot protect a patient on its own. In Texas, it was a state attorney general’s investigation, not NIST, that produced the disclosure requirements. And the RMF tells you what good outcomes look like while deliberately not telling you exactly how to build them, which is both its strength and the place where many organizations stall.

That is where Operational AI Readiness picks up. Chapter 4 turns several of these outcomes into a minimum viable control stack: inventory, named ownership, real human oversight, runtime visibility, approval gates, change control and validation. On inventory, it is blunt: an organization that cannot list its consequential AI workflows “is not yet governing AI. It is merely surrounded by it” (Bakos, 2026). Chapter 11 explains why this matters when scrutiny arrives: reviewers ask who owns the system, what changed, where the human review is, and then, “Show me the record” (Bakos, 2026).

The verdict

Does the evidence earn my proposition? On the framework, yes: Measure says test before deployment and keep testing in operation, and the Texas agreement reads like Measure and Map written into a legal settlement. On the case itself, only partly, and I want to be precise about why.

The Texas case was resolved by an agreement, not a trial. The state’s account is an allegation, and Pieces denies wrongdoing; the public record we read does not establish the tool’s actual error rate. Nor does it name the hospitals, describe how their staff used the summaries, or report that any patient was harmed. Nothing we read suggests either side was working to the NIST framework, which is voluntary. We use the case to show what the framework’s questions look like in practice, not to judge anyone against it.

So the verdict is this: the framework will not stop a bad number on its own, but it gives everyone, from a hospital board to a patient’s daughter, the right words to refuse one. You do not need to cite subcategory numbers to ask an organization: what is this system for, how did you test it, and who can switch it off?

What we still don’t know

We don’t know what the revised framework will change, or whether the Texas approach of making vendors explain their accuracy figures will spread to other states. And here is the question I can’t yet answer: how many AI tools writing about patients today were bought on a number nobody outside the vendor has ever checked?

Questions you can ask

  • If AI is used in your care or your case: Is an AI tool summarising or drafting records about me, and does a person check what it writes?
  • If AI is used in your care or your case: If I find a mistake in a record, how do I get it corrected?
  • If your organization uses AI: For our most consequential system, can we answer the four verbs today: who governs it, what is it for, how is it measured, and who can switch it off?
  • If your organization buys AI: What exactly does the vendor’s accuracy figure count, and have we measured it ourselves, in operation, on our own work?

Go further. Chapter 4 of Operational AI Readiness turns frameworks like the NIST AI RMF into the smallest set of controls that actually works, and Chapter 11 shows what auditors, regulators and internal reviewers will ask to see. Read it free, or test one workflow with the free readiness assessment.

Your next step: read Operational AI Readiness free. No account, no email wall. Start with Chapter 4 and keep one real system in mind as you read.


FairByDesign is reader-funded. If this helped, get the free Mercury Brief or see ways to support. Support never buys editorial control.

References

Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live. FairByDesign. https://fairbydesign.org/operational-ai-readiness/

National Institute of Standards and Technology. (2023). Artificial intelligence risk management framework (AI RMF 1.0) (NIST AI 100-1). U.S. Department of Commerce. https://doi.org/10.6028/NIST.AI.100-1

National Institute of Standards and Technology. (n.d.). AI risk management framework. Retrieved October 15, 2026, from https://www.nist.gov/itl/ai-risk-management-framework

Texas Attorney General. (2024a, September 18). Attorney General Ken Paxton reaches settlement in first-of-its-kind healthcare generative AI investigation [Press release]. https://www.texasattorneygeneral.gov/news/releases/attorney-general-ken-paxton-reaches-settlement-first-its-kind-healthcare-generative-ai-investigation

Texas Attorney General. (2024b, August 21). Assurance of voluntary compliance, State of Texas v. Pieces Technologies Inc. [Petition for approval, file-stamped]. https://texasattorneygeneral.gov/sites/default/files/images/press/Petition%20for%20Approval%20of%20AVC%20Pieces%20File%20Stamped.pdf

Leave a Reply