Co evolutionary learning defends language agents from hidden attacks
CoRL: Co-Evolutionary Reinforcement Learning for Adaptive Indirect Prompt-Injection Attacks and Defenses
Machine Learning
Summary
Language agents that use external tools can be tricked by hidden instructions inside the tools' outputs, which changes what the agent does without obvious signs. The authors study this problem as a game between an attacker and a defender, where the attacker adapts its hidden messages and the defender tries to block them while completing the tasks. They develop CoRL, a three-step method that trains both attacker and defender together and improves defense over fixed attacks. Their experiments show CoRL stops nearly all attacks and keeps the agents working well. This approach also helps create better adaptive tests for these defenses.
What this means in practice
- •For software security teams: Protect language-based software that uses external tools from hidden input attacks adapting over time using co-evolutionary training.
- •For ai safety engineers: Develop more robust language agents that maintain task goals even under stealthy adversarial instructions in tool outputs.
Authors
Boyang Zhang, Qingxin Xiao, Lingwei Dang, Qingyao Wu
Abstract
Tool-augmented language agents are vulnerable to indirect prompt injection (IPI). Unlike direct prompt injection, IPI hides adversarial instructions in untrusted tool outputs and can covertly alter the execution of a legitimate task. Defenses trained on fixed attacks may fail as an attacker changes its strategy, injection site, and payload. To address this problem, we formulate adaptive IPI as an asymmetric, partially observable, general-sum Markov game: a multi-turn attacker adapts payloads at reached tool-return sites from the public trajectory, while a tool-using defender must block the injected objective and complete the user task. We propose CoRL, a verifier-grounded co-evolution and repair framework with three stages: Attacker SFT initializes multi-turn attacks from successful trajectories; bilateral Co-PPO jointly trains both agents with role-specific rewards and historical opponent populations; and Defender SFT consolidates verifier-accepted teacher repairs for population-discovered failures. Across 1,514 clean, fixed-template, and adaptive executions per defender, CoRL reduces overall ASR by 38.5 points to 0.0% and raises utility by 13.1 points to 76.3%. Stage-wise and controlled ablations show positive contributions from online Co-PPO and population-mined repair, while external-benchmark evaluation indicates transfer in attack resistance. The defender balances safety and task utility under the evaluated attacks, while the retained attackers provide candidates for adaptive red-team evaluation.