FairByDesign Evidence Lab
We test whether AI security and governance claims survive contact with running systems.
FairByDesign Evidence Lab is the empirical research arm of FairByDesign. It builds controlled test environments, instruments them, attacks them, measures what happens, and publishes enough of the method for others to inspect, challenge and reproduce the result.
The Lab exists because policies, standards, product claims and architecture diagrams do not prove that a control works.
A security claim becomes useful when we can define what it means, construct a fair test, collect evidence, expose failures, and rerun the experiment as the technology changes.
Current research
The debut study.
Evidence Lab 001 — The Agent Control Gate
Can runtime controls stop unauthorized AI-agent tool actions?
The debut experiment compares an uncontrolled agent, prompt-only restrictions, runtime policy enforcement and runtime enforcement with explicit approval requirements. It tests direct misuse, indirect prompt injection, chained tool actions, revoked authority, ambiguous approvals and repeated attempts. The study also measures false blocks, enforcement overhead and whether the resulting evidence can reconstruct the incident.
What we investigate
Seven areas where claims about AI control are easy to make and hard to verify.
Agentic security
We test the security properties of systems that plan, call tools, use memory, retrieve external content, delegate work and take actions. Research may include prompt injection, tool misuse, privilege boundaries, memory and context poisoning, action chaining, excessive agency, approval bypass, compromised dependencies and multi-agent behavior.
Agent control standards and runtime enforcement
We examine whether proposed control mechanisms can actually constrain agent behavior. This includes the OWASP Agent Control Standard (ACS), middleware and policy enforcement, authorization boundaries, approval gates, runtime interception, observability and portable control patterns.
Security claims
We turn statements such as “secure by design,” “enterprise-grade governance,” “human in the loop,” “least privilege,” “auditable,” “traceable” or “policy enforced” into testable propositions. When a claim cannot be operationalized, we say so. When the evidence contradicts the claim, we publish that too.
MCP and tool security
We study Model Context Protocol and other agent-to-tool interfaces as security boundaries: authorization, tool discovery, routing, permission scope, malicious or misleading tool content, approval flows, state, identity, evidence capture and failure handling.
Evidence, reconstruction and accountability
A system can emit logs without providing enough evidence to establish what happened. We test whether an incident can be reconstructed: what the agent did, why it did it, what influenced it, which controls were active, whether intervention was possible and who or what had authority.
Human oversight
We test whether a human can actually understand, stop, correct or reverse an agent’s behavior under realistic conditions. “Human in the loop” is treated as a measurable control, not a reassuring phrase.
Longitudinal change
AI systems, models, frameworks and protocols change. Selected experiments are rerun over time so a result does not quietly become obsolete.
The FairByDesign research standard
Every Evidence Lab study follows the same core sequence.
State exactly what is being tested and what would count as support or contradiction.
Record the hypothesis, threat model, variables, scenarios, metrics, exclusions and stopping rules before the main run.
Use code, controlled systems, versioned configurations and structured logging. No production systems are attacked without explicit authorization.
Run benign and adversarial dry tests to verify that the experiment measures what it says it measures.
Preserve run IDs, versions, inputs, control decisions, tool actions, outputs, timing and failure evidence.
Report failures, false positives, uncertainty, outliers and limitations. Averages do not get to hide important failure modes.
Publish the methodology and, where safe and practical, code, schemas, synthetic datasets and reproduction instructions.
Re-measure important studies when frameworks, models, controls or standards materially change.
What every lab publication includes
So another researcher can see how a result was produced and where it might be wrong.
- Research question and claims under test
- Threat model and scope
- Test environment and exact versions
- Scenario definitions and expected outcomes
- Control configurations
- Measurement definitions
- Raw-result or structured-result tables
- Failure cases and counterexamples
- Reproduction instructions
- Known limitations and unresolved questions
- Change log and rerun history
How the Lab works
The rules that keep the evidence honest.
How AI is used
AI may accelerate literature review, code scaffolding, test generation, debugging and exploratory analysis. It does not get to manufacture evidence. Material claims are checked against primary sources or generated by reproducible experiments. Results are derived from recorded runs, not from an AI model’s description of what probably happened.
Independence rule
The Lab does not exist to make FairByDesign, a vendor, a framework or a preferred thesis look good. An inconvenient result is still a result. If a control fails, the failure is reported. If a hypothesis is unsupported, the paper changes rather than the data.
Responsible testing
FairByDesign Evidence Lab uses controlled, simulated or explicitly authorized systems. Research is designed to improve defensive understanding, control effectiveness and accountability. Sensitive exploit details may be withheld when publication would create unnecessary risk.
Publication cadence
One deep study each month
These studies are slower than ordinary commentary because the rig, data and reproduction path matter as much as the prose.
Living benchmarks
Selected studies may become living benchmarks and be rerun quarterly or after material model, framework, protocol or control changes.
Research foundations
The Lab tracks and tests emerging work including the OWASP GenAI Security Project and Agent Control Standard, the OWASP Top 10 for Agentic Applications, NIST AI evaluation and agent-security work, the Model Context Protocol specification, and relevant security, privacy and AI governance standards.
Standards and guidance are inputs to experiments, not substitutes for evidence.
FAIRBYDESIGN
Independent intelligence and reproducible research for a world being rewritten by AI.