Persistent memory attacks on AI agents cause lasting harm
From Attack Success to Attack Severity: Counterfactual Memory Attacks on LLM Agents
Artificial Intelligence
Summary
Large language model agents keep memory from past interactions to improve conversations, but this memory can be hacked to influence their future actions. The paper studies not just whether an attack works, but how much damage the attack causes later on. The researchers created MemHarm, a method that finds memory attacks causing bigger downstream problems by focusing on how harmful changes are, rather than just if they succeed. They show MemHarm outperforms other attacks in creating more serious consequences for the agent’s behavior.
What this means in practice
- •For ai security teams: Detect and defend against persistent memory attacks that cause long-lasting harm to AI agents.
- •For software engineers: Design safer LLM agents by evaluating attack severity using counterfactual memory regret to limit harmful memory writes.
Authors
Mingxi Zou, Langzhang Liang, Zhuo Wang, Yiyang Zhao, Lizhen Qu, Zenglin Xu
Abstract
As LLM agents increasingly rely on persistent memory for long-horizon and personalized behavior, they can retain and reuse information across interactions, but this also creates a lasting channel through which malicious memory writes can influence future behavior. Persistent-memory attacks are typically evaluated by whether they succeed, yet successful attacks can leave persistent states with substantially different downstream consequences. We study this severity as a distinct attack-design objective and formalize it with counterfactual memory regret (CMR), the paired increase in expected downstream loss relative to clean memory. We introduce MemHarm, which predeclares a finite class of sparse, grounded semantic edits, evaluates candidates through the normal agent memory interface using offline paired-loss feedback, and certifies resolved selections within that class. Compared with attack-success optimization, CMR-guided selection produces substantially larger downstream loss while retaining most of the success-rate gain. Across two agent benchmarks and diverse memory designs, MemHarm attains the highest CMR point estimates among the evaluated general attacks on identical support. Factor-removal interventions link this harm to the selected semantic factor, and native-agent deployments verify the write-to-fresh-process attack path.