Agent security risk rises when users and AI share control
DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security
Artificial Intelligence
Summary
Large language model agents often work by interacting with people and other tools, but most security tests assume the user just watches and does not affect the system. The authors created DUMA-Bench, a test setup that checks how safe these agents are when both the user and the agent can change what’s going on. They found that when users can also control parts of the environment, attacks against agents succeed much more often. The study shows that keeping AI safe depends on the back-and-forth between the AI, the user, and the environment, not just the AI model itself.
What this means in practice
- •For ai security teams: Evaluate and improve security of AI agents by testing how they handle interactions with users who can influence shared environments.
- •For chatbot platform developers: Use the benchmark to identify vulnerabilities in multi-agent systems and design safer user-agent interaction protocols.
Authors
Ivan Aleksandrov, German Kochnev, Sabrina Sadiekh, Yaroslav Rogoza
Abstract
LLM-based agents increasingly operate in environments where they interact with users, tools, and external systems. Yet most security evaluations assume passive users and static control, ignoring the interactive dynamics that shape real agent behavior. We introduce \textbf{DUMA-Bench}, a benchmark and evaluation protocol for measuring agent security under \emph{dual-control} interaction, where both the agent and the user can influence the shared environment state. DUMA-Bench extends $τ^2$-bench ~\cite{barres2025tau} with adversarial environments covering eight vulnerability classes, including RAG poisoning, cross-agent manipulation, and unsafe output handling. We evaluate \textbf{14 models from five model families} (OpenAI, Anthropic, DeepSeek, Qwen, and Z.ai) across eight domains and multiple user-behavior regimes. Across our experiments, introducing dual-control interaction increases the attack success rate from \textbf{26.9\%} to \textbf{41.1\%}. These results show that agent security is not solely a property of the model but emerges from the interaction between the model, the user, and the environment. DUMA-Bench provides a missing evaluation layer for studying security in realistic agent deployments.