Untrusted data must not steer what your agent does.
ProvGuard sits between an LLM agent and its tools. It does not try to spot malicious text. It checks where every sensitive argument came from, and refuses the call when only an outsider supplied it.
F1 of a fine-tuned prompt-injection detector when the test data comes from a different source. Detection does not transfer, as my detector benchmark shows.
AgentDojo banking attacks stopped when replaying the attacker's own reference calls, with policies written before any attack was read.
Attack success of a live Qwen3 8B agent on AgentDojo banking, without and with ProvGuard, while it kept solving the same share of tasks.
Hijack an agent, then watch the guard decide
Edit the user's request, the e-mail the agent read, and the tool call the model makes. This is the same algorithm as the Python library, ported to JavaScript.
Policy: send_email.to and transfer_money.iban / amount require a trusted origin. Once untrusted text is in context, send_email needs confirmation and transfer_money is blocked.
What it stops, and what it costs
AgentDojo ships a reference solution for every task. Each user task runs in a poisoned environment, then the attacker's reference calls follow as if the agent were fully hijacked. No model is in the loop, so every number is deterministic and reproducible with one command.
| Suite | Attacks stopped | Pairs | Legitimate tasks blocked |
|---|
Written blind, fixed in the open
The four policies were committed before any injection task was read and have not changed since. The first run exposed one real flaw: in Slack the attack text sat inside a channel name, and the channel list was trusted, so a URL inside that name looked trusted too. v2 changed the library, not the policies. Structured tool output now vouches only for complete field values, such as the channel general, never for text found inside a field.
What still gets through
- Attacks that pick among options that are legitimate anyway: book the priciest hotel in the real catalogue, delete a file id from the user's own list.
- Data sent to a recipient the user really named, inside the message body.
- Writes with no sensitive argument, such as a calendar event with no guests.
Argument provenance cannot stop these by design. Confirmation on state-changing tools moves them to a person. The benchmark has no person, so the gap is reported instead of hidden.
A live agent
Qwen3 8B running locally, AgentDojo banking: all 16 user tasks, and the first three injection tasks against each of them. Same policy, same attack, one run per defence at temperature 0.7, so one-task differences in utility are noise. With 0 of 48, the one-sided 95% upper bound on attack success under ProvGuard is about 6%.
How a decision is made
Label every input
User and system messages are trusted. Tool output is untrusted unless the policy says nobody else can write into it.
Trace each argument
Before a tool runs, each argument value is matched against the labelled inputs: whole tokens, case and spacing ignored.
Require a trusted origin
A protected argument passes only if a trusted source contains it. Values from e-mails, web pages, or the model's imagination fail closed.
Escalate when tainted
Once untrusted text is in context, sensitive tools can require confirmation or be blocked outright.
version: 1
default:
on_tainted: confirm # unknown tools never pass silently
tools:
read_inbox:
output: untrusted
get_contacts:
output: trusted
send_email:
on_tainted: confirm
args:
to:
require: trusted
on_violation: block
Limits, stated up front
- Provenance is value matching, not token-level tracking. A value that a trusted input happens to contain will pass.
- Free-text arguments such as an e-mail body are not constrained, so leaking data through them remains possible.
- The guard only knows what it is shown. Input that bypasses it is invisible to it.
- Under strict policies, tasks that delegate a choice to untrusted content ("pay the bill in this file") need confirmation.