Self evolving AI agents face new hidden backdoor attack risk
EvolvAttack: Backdooring self Evolving VLM Agents
Machine LearningArtificial Intelligence
Summary
This paper shows that some AI agents, which change and learn over time, can be secretly hacked in a way that doesn’t show up at first. The hackers create hidden triggers that only activate after the AI has evolved enough, so the attack is very hard to detect by normal tests. The researchers demonstrate this sneaky attack on several types of vision-language AI agents, where the bad behavior jumps from almost nonexistent to very effective after about 100 interactions. This reveals a unique security challenge for evolving AI systems that previous studies on static models didn’t address.
What this means in practice
- •For ai security teams: Detect and defend against evolving backdoor threats in adaptive vision-language agents used in deployed AI systems.
- •For software developers: Design testing strategies for AI products that include checks for backdoors triggered by agent memory state changes over time.
Authors
Chuan Zou, Yang Xiang, Zexin Li, Xiaonan Luo, Simin Chen
Abstract
Self-evolving mechanisms let vision-language model (VLM) agents continually adapt through evolving memory, but they also allow malicious behavior to emerge only after deployment. We introduce a new attack surface, the self-evolving backdoor, whose payload stays dormant in the initial agent and is progressively activated as the agent evolves. Unlike conventional backdoors, which can be exposed by testing triggered inputs on a static model, our attack requires both a visual trigger and a sufficiently evolved memory state; static scanning with clean memory thus yields near-zero attack success and may deem the agent benign. We realize this through memory-aware backdoor training, which couples trigger activation with the agent's evolving memory while preserving benign behavior unless the trigger co-occurs with payload-related memory. Across three VLM backbones, four agent pipelines, and recency-, semantic-, and reward-based memory evolution, the attack preserves normal functionality and stays nearly inactive before evolution (0.5% attack success rate), yet exceeds 95% attack success within 100 interactions, revealing a stealthy threat unique to self-evolving agents.