Is Your AI Really Just a Suggestion? Classifying Decision-Influencing AI

From the Field Manual. Each week we take one chapter of our free book Operational AI Readiness and turn it into something you can use. This week: Chapter 3, on classifying AI honestly.

Almost every AI system at work arrives with a reassuring label: assistant, copilot, decision aid. The label is usually sincere. The question this chapter asks is whether it tells you anything about what the system actually does to the decisions around it.

Consider a tool that has been described with unusual care. Since August 2016, Allegheny County in Pennsylvania has used the Allegheny Family Screening Tool when its child welfare hotline decides whether a report of possible neglect should be investigated (Allegheny County Department of Human Services [ACDHS], 2018, p. 3). The model draws on more than 100 factors to produce a score from 1 to 20, and call screeners see the highest score among the children on a referral (ACDHS, 2018, p. 4). The county has been clear about its intent: “It was never intended or suggested that the algorithm would replace human decision-making” (ACDHS, 2018, p. 4). Supervisors can override the score, and the county says overrides are documented and reviewed (ACDHS, 2018, p. 9).

Now look at how the tool presents itself on screen. Some scores were labelled “mandatory screen-ins” (ACDHS, 2018, p. 9). By the county’s 2024 description, a score of 18 to 20 with a child under 16 in the home shows the screener a “high-risk protocol” instead of a number, and the protocol defaults the screening decision to investigate, though supervisors override the defaults “and do so regularly” (ACDHS, 2024, pp. 2–3). A default is not a decision. But anyone who has ever accepted a pre-ticked box knows it is not nothing, either.

The scrutiny has been real. In April 2022, according to the Associated Press, Carnegie Mellon researchers found that in its first years the tool flagged a disproportionate number of Black children for “mandatory” neglect investigation, and that social workers disagreed with its scores about a third of the time; the county called the research “hypothetical” and stressed that workers can override the tool (Ho & Burke, 2022). In January 2023 the AP reported that the Justice Department’s Civil Rights Division was examining the tool after complaints about how disability-related data could affect families; the department declined to comment (Ho & Burke, 2023).

The chapter’s core idea: influence is not the same as authority

We are not using Allegheny as a villain. It published more about its tool than most organizations ever do. We are using it because it shows the exact problem Chapter 3 of Operational AI Readiness was written for. Viktoria Bakos puts it bluntly: “A workflow does not become low consequence just because the team gives it a soft name” (Bakos, 2026, p. 29). A human may hold the final button, but “human presence does not erase AI influence” (Bakos, 2026, p. 29). A ranking decides what gets looked at first. A summary decides what a reviewer notices. A default decides what happens when someone is tired.

The chapter sets out four principles. Influence counts even when decision authority stays human. Classify by what the workflow can affect, not by what the model technically outputs. Context of use outranks the generic product description. And reasonably foreseeable misuse belongs inside classification, not in a footnote, because tools drift: a summarizer becomes a denial-letter drafter, an analyst aid becomes the main triage mechanism (Bakos, 2026, pp. 29–31).

Five questions that resist euphemism

The practical heart of the chapter is a short test. Does the system influence a decision or action, rather than help with low-stakes drafting? Can that influence affect money, access, employment, eligibility, security, legal position, safety or rights? Can it trigger, recommend or speed up downstream actions, even indirectly? Is it foreseeable that people will over-rely on it because of speed, workload or confidence? And if something went wrong, would you need to reconstruct the workflow in detail to explain yourself? When the answers cluster around yes, the system belongs on a stronger readiness path (Bakos, 2026, p. 32).

Run Allegheny’s public description through those questions and every one points the same way. That does not mean the tool should not exist. It means its honest classification is “consequential,” and controls like the ones the county describes, such as documented overrides and keeping the score away from investigators (ACDHS, 2024, p. 4), are what that classification should demand. Note too that operational classification is not a legal determination; it should be broader and faster (Bakos, 2026, p. 32).

What your team can do this month

• Answer the five questions in writing for every AI tool described as an assistant, copilot or aid. Treat “not sure” as yes until someone can show otherwise.
• Write the three statements the chapter says classification must produce: intended purpose and excluded uses, the domains of consequence the system can touch, and whether it belongs on a stronger control path (Bakos, 2026, p. 33). If you cannot write them without hand-waving, the work is “just socially padded.”
• Hunt for defaults and labels. Anywhere the AI pre-fills a choice, ranks a queue or tags something “low risk,” you have influence. Count how often people override it; Allegheny reported its own override pattern publicly (ACDHS, 2018, p. 10).
• Ask frontline users what they really use it for. The gap between the approved use and the actual use is where foreseeable misuse lives.
• Start controls before the legal answer arrives. Waiting for legal certainty, the chapter argues, confuses legal exposure with basic operational discipline. Let counsel work in parallel.

What this means for ordinary people

If an organization tells you a decision about you was made by a person, that may be true and still incomplete. The AP reported that in Allegheny, families and their lawyers are not allowed to know the scores (Ho & Burke, 2022). You can still ask whether any automated score, ranking or summary was part of the process, and whether a person reviewed it with the power to disagree. “A human decided” should be the start of the explanation, not the end of it.

What the record does not show

The documents we read do not show whether the tool improved or worsened decisions overall; the research is contested and the county disputes critics’ findings. We found no public outcome of the Justice Department’s inquiry. The county’s descriptions are dated 2018 and 2024 and the tool has been revised over time, so we cannot say exactly how it works today. Nothing here suggests that any individual screener acted improperly.

Questions you can ask

• If you are affected: “Was any automated score, ranking or summary used in my case, and could the person reviewing it overrule it?”
• If you are affected: “Where can I see what information about me went into that score?”
• If you run AI at work: “Which of our ‘assistive’ tools set a default, rank a queue or draft the words of a decision?”
• If you run AI at work: “Could we show today how often reviewers disagree with the tool, and what happens when they do?”

Your next step: take the free readiness assessment on one AI tool you use. It opens with the same five classification questions, where “not sure” counts as yes, then shows your weakest control domain, three next moves and the chapter to read. Nothing is stored; it is a self-assessment, not legal advice or an audit.

---

FairByDesign is reader-funded. If this helped, get the free Mercury Brief or see ways to support. Support never buys editorial control.

References

Allegheny County Department of Human Services. (2018). Allegheny Family Screening Tool: Frequently-asked questions (Updated August 2018). https://www.alleghenycountyanalytics.us/wp-content/uploads/2018/10/17-ACDHS-11_AFST_102518.pdf

Allegheny County Department of Human Services. (2024, April). Summarizing recent research on predictive risk models in child welfare. https://www.alleghenycountyanalytics.us/wp-content/uploads/2024/05/24-ACDHS-04-Predictive-Risk-Algorithms.pdf

Bakos, V. (2026). Operational AI readiness: A practitioner guide to governing AI that is already live (Free ed.). FairByDesign. https://fairbydesign.org/operational-ai-readiness/

Ho, S., & Burke, G. (2022, April 28). An algorithm that screens for child neglect raises concerns. Associated Press (via KTVZ). https://ktvz.com/news/2022/04/28/an-algorithm-that-screens-for-child-neglect-raises-concerns-2/

Ho, S., & Burke, G. (2023, January 31). AP report: DOJ examining AI screening tool used by Pa. child welfare agency. Associated Press (via PBS NewsHour). https://www.pbs.org/newshour/amp/nation/ap-report-doj-examining-ai-screening-tool-used-by-pa-child-welfare-agency