Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents
2026-08-03 • Cryptography and Security
Cryptography and Security
AI summaryⓘ
The authors study self-improving AI agents that learn by saving and reusing past experiences. They find a new safety risk where harmless experiences, when combined over time, can weaken the agent's safety and cause problems. To explore this, they created EvoBreak, an attack that subtly manipulates sequences of safe experiences to trigger unsafe behavior. They also built BreakGym, a tool to train and test such attacks. Their experiments show EvoBreak is more effective and stealthy than previous methods, highlighting the hidden dangers in how these AI agents learn from their own past.
Self-evolving agentsLarge language modelsExperience distillationMemory attacksSafety boundarySequential attackRejection samplingFine-tuningDependency structuresSafety benchmarks
Authors
Bingyu Yan, Xiaoming Zhang, Chaozhuo Li, Ziyi Zhou, Yirui Qi, Litian Zhang
Abstract
Self-evolving large language model agents improve their capabilities by distilling interaction trajectories into persistent experiences. Yet this mechanism introduces a new safety risk: experiences that are benign in isolation may jointly weaken an agent's safety boundary when accumulated and reused across sessions. Existing memory attacks typically require direct memory access or induce explicitly malicious records, limiting their stealthiness and applicability. We propose EvoBreak, an experience-conditioned sequential attack that operates through individually benign attack-stage tasks and induced experiences. EvoBreak repeatedly observes the experiences distilled by the victim, identifies uncovered target-relevant requirements, and adaptively acquires complementary experiences before reformulating the final query to activate them jointly. To support training, we introduce BreakGym, a structure-first synthesis pipeline that generates decomposable safety-sensitive targets with diverse dependency structures. EvoBreak is optimized using rejection-sampling supervised fine-tuning and Hint-guided GRPO. Experiments across self-evolving frameworks, victim backbones, pre-evolution domains, and safety benchmarks demonstrate that EvoBreak consistently outperforms existing attacks while maintaining high benignness. These results reveal benign experience composition as a persistent attack surface in self-evolving agents.