Agent safety improves with attacker defender co evolution training
AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety
Artificial Intelligence
Summary
Large language model agents use tools to complete tasks, but harmful inputs can trick them into unsafe actions. The authors developed AgentSafeEvo, a training method where an attacker creates difficult challenges to expose the agent’s weak points, and a defender learns to safely complete tasks despite these attacks. This back-and-forth training helps the agent become safer without losing its ability to do other tasks well. Their experiments show this approach strengthens the agent’s defenses against manipulation.
What this means in practice
- •For ai platform engineers: Improve safety of AI agents by training them with adversarial tasks to resist malicious input manipulation.
- •For security teams: Test and harden AI tool use against prompt injections and adversarial attacks through co-evolutionary training.
Authors
Chenxu Wang, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Cheng Jin, Chaochao Lu, Qiaosheng Zhang
Abstract
Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.