What was measured, and what it does not show

Five benchmarks are written up in full on this page. Some of them make this product look good and some of them do not, and they are all here for the same reason: a detection claim only the vendor can reproduce is not evidence. Every number on this page is copied from a results file checked into the repo, and the file is named under each table.

These five are not all of them. The benchmarks/ directory in the repository holds sixteen directories with a README of their own, including answerability, composed_privilege_escalation, crescendo, entitlement, pii, redteam and source_authority. Each has its own README and, where a number exists, a results file checked in beside it; a few record why no benchmark number applies to the mechanism at all rather than forcing one. benchmarks/README.md is the index. The five below are the ones this page writes up, not the ones that came out best.

1. Containment when detection has already failed

This is the benchmark the product actually rests on, and it starts by assuming the detectors lose. The Attacker Moves Second (Nasr, Carlini, Schulhoff et al., 2025, arXiv:2510.09023) reports over 90% attack success against twelve published defences once the attacker is allowed to adapt. Our injection detectors are not an exception to that, so measuring detection alone answers the wrong question. The question is what happens after the model has been convinced.

Every scenario runs twice through the real Enforcer.guard_tool_call path, the same call the SDK, the LangGraph nodes, the MCP governor and the gateway all make before a tool executes. In the second run NOMETRIA_ENABLED_DETECTORS is set to the empty list. That is a total bypass, not a weakened threshold or a simulated miss, and it is checked rather than assumed: each scenario's payload is re-run through check_content in the same mode and the entity count recorded. Across the whole attack set in that mode, detector entities raised: 0.

The agent is then treated as fully compromised. It attempts precisely the action the attacker's text asked for, with argument values taken from that attacker-controlled text. What is being measured is whether anything stops it that never read the content at all: capability grants, argument-provenance taint ceilings, declared numeric constraints, generated-statement analysis, cascade analysis, and the kill switch.

Source: benchmarks/containment/README.md, results table. Raw per-scenario output is in benchmarks/containment/results/containment_results.json.

The eight scenarios, with detection fully disabled

Source: benchmarks/containment/README.md, per-scenario table.

The four negative controls, also with detection disabled: a $20 in-ceiling refund, a knowledge-base search carrying retrieved taint, an ordinary ticket write and a customer lookup all proceeded normally. A product that contained everything would be useless, so a blocked control is scored as a failure of this benchmark rather than a success.

What this containment result does not show

• It is not a claim that our detection is good. It is the opposite. The benchmark is only meaningful because detection is assumed to have failed completely. The detection numbers are in sections 4 and 5 and they are considerably less flattering.
• It does not measure whether a model can be convinced. The compromised agent is a premise here, not a finding.
• Containment is exactly as good as the declarations behind it. Grants, tool impact tiers, numeric constraints, trigger declarations and access scopes are all operator-declared. An irreversible tool declared as read, or an undeclared downstream trigger, is invisible by design. Scenario cb5 is contained partly because the seed data declares that tickets.update fires the helpdesk's email.send webhook. An undeclared trigger would not be seen.
• A high-risk agent escalates every irreversible action, by policy. The shipped EU AI Act pack's eu.art14.human_oversight rule sends any irreversible action by a risk_tier: high agent to a human regardless of provenance. That is why a legitimate transfer by payments-ops is not used as a negative control: it escalates by design, and scoring that as over-blocking would be dishonest in the other direction.
• Eight scenarios is a small set. It covers one instance of each containment mechanism, chosen to be structurally different from one another rather than to inflate a denominator.

One more result worth keeping: the detectors-on entity count fell from 15 to 10 on 2026-09-16, and the containment column did not move. The five that disappeared were an artefact, a curly apostrophe or an em dash being counted as a homoglyph, so ordinary typography was raising an obfuscation signal. Removing it lost detections that were never real and changed nothing about what was contained.

Source: benchmarks/containment/README.md, results and "What this benchmark does not show".

2. The same claim at scale: an AgentDojo replay of 617 calls

The eight scenarios above are ours. This one is not. AgentDojo (MIT, ETH Zurich) is the reference dynamic benchmark for prompt injection against tool-using agents, and it ships hand-authored ground-truth call sequences for both halves of its scenarios. That means no model is needed and the replay is deterministic and offline: 552 calls a correctly-behaving agent makes for its real assignment, and 65 calls a successfully compromised agent makes on the attacker's behalf. 617 calls in total, each replayed through the same Enforcer.guard_tool_call path as everything else on this page. User-task arguments are marked user-sourced; injection-task arguments are marked as arriving from tool output, which is AgentDojo's own threat model.

The setup is deliberately strict against us. One agent per suite is granted exactly the tools its own legitimate user tasks call, at max_taint: user, with no blanket approval requirement, so an injection call is contained by provenance, impact and constraint logic rather than because a grant it needed was conveniently withheld. The agents are risk_tier: limited on purpose: the shipped EU AI Act pack escalates every irreversible action by a high-tier agent regardless of provenance, which would have made containment complete for a reason unrelated to the attack.

Source: benchmarks/agentdojo_e2e/README.md, results table. Raw per-call output, all 617 calls in both detector modes with verdict and rules fired, is in benchmarks/agentdojo_e2e/results/agentdojo_e2e_results.json. Per suite, injection calls contained: banking 21/23, slack 12/13, travel 17/17, workspace 12/12.

The two columns are identical, and that is the finding. Detection contributed nothing here, because an injected send_money call is syntactically ordinary and there is no malicious string in its arguments to catch. Everything that stopped these attacks was provenance, impact tier and grant logic. The five rules that fired most often across the run were capability.approval_required (57), taint.irreversible_tool (37), intent.undeclared_irreversible (37), taint.write_from_tool_result (5) and capability.denied (5).

The 95.4% does not travel without its denominator

All three escapes are read-only calls: get_scheduled_transactions twice, and get_channels, each of them allowed. The compromised agent was told to read something it already held a legitimate grant for, and it did. Nothing in impact-tier or taint logic distinguishes that from ordinary work, by design, and a product that blocked those reads would also block the agent doing its job, which is what the 552/552 column measures. The harm in that attack shape arrives later, when the data leaves, which is the irreversible-tool step this run does contain.

What this replay does not show

• It is not AgentDojo's "utility under attack" metric. That requires driving a live model through the environment, which needs a model and a network. The two halves that can be measured exactly are measured; the one that cannot be done offline is skipped rather than approximated.
• The impact tiers are our judgement, not AgentDojo's. AgentDojo has no impact model. Ours is applied mechanically by verb (send_money, delete_*, send_email, reserve_* and similar are irreversible, create_* and update_* are writes, get_*, search_* and read_* are reads) so a reader can check it. This is the most load-bearing assumption in the run: a tool mis-declared as read is not contained.
• No model was fooled to produce these calls. The compromise is the premise, taken from AgentDojo's own answer key, not something this run demonstrates.
• 552 user calls is a utility check, not a quality check. It shows governance did not block legitimate work. It says nothing about whether the agent's answers were any good.

Source: benchmarks/agentdojo_e2e/README.md, method, "What escaped" and "What this benchmark does not show". Call data: benchmarks/action_safety/data/agentdojo_calls.json, recorded in the results file as AgentDojo v1, MIT, ETH Zurich.

3. Four agent-runtime tiers, against a real llm-guard install

These four harnesses test a narrower claim: that a prompt-injection text scanner evaluates one string at a time, in isolation, with no memory of the conversation and no visibility into what tool the model is about to call. The comparison is against an actual, separately installed llm-guard and every figure attributed to it came from a real PromptInjection().scan() call, not from an asserted number.

Tier B, indirect injection via tool output

20 cases on the identical 20 strings: 10 real indirect-injection shapes (instructions hidden in an HTML comment, fake "AI processing note" framing, a seeded poisoned MCP tool description) and 10 benign documents using the same trigger vocabulary.

Source: benchmarks/agent_security/README.md, Tier B table.

llm-guard wins on precision here, by 15.1 points. It raised 2 false positives on the benign half against our 5. That cost is real and it comes from the round 4 ensemble backstop, which bought recall everywhere and paid for it in false positives everywhere. Whether the trade is worth making depends on what a deployment fears more.

Tiers D, C and A

• Tier D, excessive agency. 6 scenarios through the real Enforcer.guard_tool_call path using the shipped seed data. 6/6 correct. This tier also found a real bug: the kill switch was wired into preflight only, so a quarantined agent's tool calls were not actually stopped by it. Fixed, with a regression test.
• Tier C, tool parameter exploitation. 10 cases, 5 real and 5 negative controls, through capabilities the agent genuinely holds, so every capability check passes and any block comes purely from argument-value analysis. 10/10 correct. The gap this closed was real: order_id="*" used to go straight through because argument analysis only inspected values under three hard-coded key names.
• Tier A, multi-turn payload splitting. The "ignore all previous instructions" phrase split across three separate API calls, plus a negative control of three ordinary support turns. 2/2 correct. None of the three fragments fires alone; only the assembled window does.

llm-guard cannot participate in D, C or A, and this is reported as what it is rather than scored as a 0% loss for it. It has no tool registry, no capability model and no concept of an agent's grants, so Tier D is outside its design. It scans free text rather than structured tool-call arguments, so {"order_id": "*"} is an opaque JSON blob to it. On Tier A, scored per turn because a stateless scanner has no other option, it flags all three fragments individually, including "all previous" on its own. That is not multi-turn awareness, it is the over-triggering on isolated trigger words that the detection report below measures directly.

Source: benchmarks/agent_security/README.md, Tier A, C and D sections.

What this round did not attempt

• Tier B's precision cost was not addressed further. Raising the ensemble's threshold based on this result would be tuning against data the project deliberately treats as held out.
• Tier A's conversation-window check is wired into the SDK one-liner only, not the gateway's preflight path.
• No head-to-head cost or latency comparison. This suite measures detection and enforcement correctness, not throughput.

4. Detection on its own, which is the least flattering number here

On the primary dataset, deepset/prompt-injections (662 labeled examples), the full stack of heuristic plus classifier plus similarity reaches 66.7% recall at 100.0% precision on the held-out split of 116, up from an unmodified regex detector's 0%. Held-out is the number to trust, because the heuristic's patterns were tuned by reading the train split's false negatives.

A third of the held-out positives still slip through. Two named groups: genuinely missed attacks, including flattery-then-pivot social engineering and typo evasion such as "igmre what I said before", which slipped past every detector; and a deliberate scope boundary, since a large share of that dataset's positives are generic role-play framing ("act as a Linux terminal") with no bypass or exfiltration signal attached. Matching that definition exactly would flood real deployments with false positives on ordinary persona-based agents.

The opt-in classifier ensemble is a measured trade rather than a free win. On NotInject, 339 prompts that are entirely benign by construction and built to trigger keyword-reactive guardrails, the shipped ensemble raises 140 false positives, 41.3%. PIGuard alone raised 39, 11.5%. The secondary model is configurable and can be switched off to get the lower figure back.

Generalization, re-measured at the shipped 40ms per-detector timeout, for the heuristic-plus-classifier configuration: 85.6% recall at 92.6% precision on SPML, 98.6% recall at 100% precision on the multilingual phrase list, and 17.9% recall at 69.5% precision on in-the-wild jailbreak prompts, where the classifier times out on 99.9% of calls because those prompts average 2,156 characters. The shipped default stack, heuristic only, scores in single digits on two of those datasets. None of that is hidden and none of it is a reason to trust detection as the control that stops an attack.

Source: benchmarks/REPORT.md: headline, primary results table, "What's still missed, honestly", "The cost" under the ensemble backstop, and the round 7 table.

5. An adaptive attacker that reads our verdict and tries again

Section 4 scores the detectors against text written once by someone who never saw our output. This one lets the attacker see the verdict and the exact entity list and revise the payload, which is the protocol from the same paper section 1 cites. Each seed goes to a search-based attacker that calls the real Enforcer.check_content path, hill-climbs on the best payload so far, and picks its next mutation from a library of 33 composable operators using that feedback. Success is the strict reading: an allow effective verdict with zero entities raised. The budget is 50 attempts per search.

Seeds the stack already misses are thrown away before the search starts, because counting them would inflate this number for free. Of 81 candidates, 33 already reach allow with zero entities on the retrieved surface and 34 on direct input, leaving 48 and 47 seeds respectively: attacks the shipped stack currently stops. That is the denominator below, and it is why attempt 1 is 0% by construction. The two move sets are scored separately, because base64, hex, rot13 and fragment reassembly evade a text detector by removing the instruction from the text, while a paraphrase or another language leaves it plainly readable. Readable only is the number to quote.

Source: benchmarks/adaptive/README.md, ASR curve table, which is the asr block of benchmarks/adaptive/results/adaptive_summary.json.

Read that as: a quarter of the attacks our detectors stop are through within five adapted attempts, and roughly three in four are through within fifty, using only mutations that leave the instruction plainly readable. With the full move set every single seed falls, on both surfaces, within 25 attempts. The 73% figure quoted on the home page is the 72.9% cell in that table: the retrieved surface, readable operators only, 50 attempts. The paper's >90% result is not something we are an exception to; with the full move set we exceed it against ourselves.

165 of the 190 searches ended in a bypass. Each one was re-run on the persisting enforcement path, because the search itself runs with persist=False for speed, and 165 of 165 reproduced.

What the bypasses did once they were through

A bypass is not harm. For every bypass whose seed names an obvious harmful action, the payload that actually beat the detectors is carried into Enforcer.guard_tool_call, the same call section 1 makes, with the same shipped seed fixture and policy packs.

38 of 38 contained. A further 38 bypasses came from seeds that name no specific action ("you are now DAN", a system-prompt leak) and were excluded rather than assigned an invented one.

Source: benchmarks/adaptive/README.md, the section pairing each bypass with containment, which is the containment_pairing and pairings blocks of results/adaptive_summary.json.

What this adaptive result does not show

• It is search-based only. The paper's taxonomy also covers gradient-based attacks, RL-based attacks and human red-teaming, and found those strongest. None are implemented here, so the real-world rate should be assumed higher than this. It is a lower bound from the cheapest attacker class, not a worst case.
• A bypass is not a working attack. Success means the detector said allow and raised nothing. It does not mean a model would then obey the payload. No model is in the loop at all, so "the instruction is still readable" is an argument rather than a measurement.
• It measures the default stack only. The five detectors enabled in the run are injection.heuristic, pii.native, secrets.native, safety.lexicon and schema.json. The opt-in classifier ensemble that produces the 66.7% held-out recall in section 4 is not running, because it needs a model download and this benchmark is offline by contract. This number is not evidence about how much harder that ensemble would be to beat.
• 48 seeds is a small set and a biased one. It is specifically the subset our detectors catch, so it answers "how robust is what we do catch", not "how safe is the product against this corpus". The second question is what section 4 and section 1 are for.
• The containment column is invariant to the payload, by design. The action path never reads the attacker's text. So 38/38 is not evidence that containment resists these particular bypasses, it is evidence that containment does not depend on the bypass at all. Its limits are the declaration limits listed in section 1.

Source: benchmarks/adaptive/README.md, method and "What this benchmark does not show".

What the runs record, and what they do not

A sceptic asked what was held constant, and the answer for several of those things is "nothing was, and it is not recorded". That is written out here rather than implied away.

What the result files do record. The adaptive run records its random seed (20251009), the 50-attempt cap, 190 searches, 5,593 attempts, 29.2 seconds of wall clock, the five enabled detectors by name, and the fact that it makes no model calls and no network calls. The AgentDojo run records the source of its call data and the verdict and rules fired for each of the 617 calls in both detector modes. The containment run records a verdict and the rules that fired for each of its scenarios. The Tier B comparison records the raw confusion matrix for both systems, 10 true positives, 5 false positives, 5 true negatives and 0 false negatives for this product against 9, 2, 8 and 1 for llm-guard, along with a count of 0 examples degraded by a detector timeout on either side.

• No run date. None of these result files carries a timestamp. The only date on this page is 2026-09-16, which is when the three detector fixes landed and the containment entity count and the generalization table were re-measured. The date each of the other runs was produced is not recorded anywhere, so this page does not state one.
• No llm-guard version. The Tier B result records only that llm-guard was installed in a separate interpreter and that its PromptInjection scanner was really called. Neither the llm-guard version nor the model it loads is written down, and the 81.8% and 90.0% figures would move with either. The only version fact recorded is the conflict that forces the separate interpreter, llm-guard's pin of transformers==4.51.3.
• No hardware. No CPU, memory or machine description appears in any of these files. That is not cosmetic here: detectors run under the shipped 40ms per-detector timeout, and a detector that times out is scored as raising nothing. The adaptive run recorded 9 such degraded attempts out of 5,593, 0.16%, touching 5 searches, on whatever machine produced it. A slower machine would record more, and would move cells in the direction that flatters the attacker.
• One run each, so no variance. Every figure on this page is a single run. None is a mean, none has an error bar, and no repeat-run spread was measured. The adaptive README is the only file that says anything about stability, and what it says is that re-running can move a cell by a couple of points because of those timeouts.

Source: benchmarks/adaptive/results/adaptive_summary.json (config, reproducibility, bypass_verification), benchmarks/agentdojo_e2e/results/agentdojo_e2e_results.json, benchmarks/agent_security/results/tier_b_results.json, and the 2026-09-16 re-measurement notes in benchmarks/containment/README.md and benchmarks/REPORT.md round 7. The absences listed above are absences in those same files.

How to reproduce any of this

Each benchmark is a self-contained script with its own throwaway SQLite database, and each writes a results file that is checked in beside it.

uv run python benchmarks/containment/run_containment_benchmark.py
uv run python benchmarks/agentdojo_e2e/run_agentdojo_e2e.py
uv run python benchmarks/agent_security/tier_b_indirect_injection.py
uv run python benchmarks/run_prompt_injection_benchmark.py
uv run python benchmarks/adaptive/run_adaptive_benchmark.py

The llm-guard comparisons need LLM_GUARD_VENV_PYTHON pointing at a separate interpreter that has it installed, because llm-guard pins transformers==4.51.3 against this project's own pinned transformers>=5. Without that variable the scripts still run and report our own numbers, with the llm-guard fields returned as null.

The AgentDojo replay and the adaptive run are offline and need no model, no network and no API key. Every one of them writes the results file named above, so a number that does not match is a bug report rather than a disagreement.