DUMA-Bench extends agent-security testing to dual-control environments where both the user and agent can change shared state. Across 14 models, eight domains and 3,920 saved runs, aggregate attack success rises from 26.9% with a passive user to 41.1% with interactive adversarial behavior. The suite covers RAG poisoning, cross-agent manipulation, unsafe output, data leakage and tool shadowing, with deterministic state assertions for most checks.
DUMA-Bench finds interactive users raise agent attack success to 41%
2026-09-21Frontier watchTRIAL
DUMA-Bench extends agent-security testing to dual-control environments where both the user and agent can change shared state. Across 14 models, eight domains and 3,920 saved runs, aggregate attack success rises from 26.9% with a passive user to 41.1% with interactive adversarial behavior. The suite covers RAG poisoning, cross-agent manipulation, unsafe output, data leakage and tool shadowing, with deterministic state assertions for most checks.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For teams red-teaming agents, a one-shot malicious prompt understates deployment risk because users can adapt after every tool result. Add multi-turn, state-changing adversaries and replayable environment assertions to security gates. DUMA-Bench is useful as a starting harness, but its 35 scenarios and partly simulated users are too small to justify a production claim by themselves.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Models | 14 models across OpenAI, Anthropic, DeepSeek, Qwen and Z.ai families |
|---|---|
| Coverage | 8 domains, 35 executable scenarios, 1,960 trials per regime |
| Attack Success | 26.9% passive-user versus 41.1% dual-control, +14.2 percentage points |
| Scoring | deterministic post-state assertions plus GPT-4o communication assertions where required |
| Judge Audit | manual review agreed with 56 of 60 sampled communication-assertion decisions |
| Code | public GitHub repository |
Technical impact 3
Engineering value 4
Adoption signal 2
Maturity 2
Verification cost 3
Risk 2