Memory formation in AI agents vulnerable to manipulation at multiple points
MemForm: Benchmarking Memory Formation Poisoning in LLM Agents
Cryptography and SecurityComputation and Language
Summary
LLM agents can remember past interactions to improve over time, but this memory-making process can be tricked or poisoned in several ways. The authors studied five ways attackers can influence how agents form and store memories, introducing new categories to better understand these weaknesses. They tested various AI models and found that all are vulnerable to memory poisoning, even when the original experiences seem genuine. Their work suggests it's important to protect both what memory is saved and how it is created.
What this means in practice
- •For ai safety teams: Identify and defend against different ways attackers can corrupt an AI agent’s long-term memory formation process to improve reliability.
- •For ai system builders: Design LLM agents with memory formation protocols that consider retention order and consolidation effects to reduce vulnerability to memory poisoning.
Authors
Haowen Pan, Pengyang Shao, Yunshan Ma, Yanzheng Jin, Naixin Zhai, Xiang Wang, Xun Yang
Abstract
Long-term memory enables LLM agents to learn from past interactions but exposes memory formation to adversarial influence. Existing studies primarily organize memory poisoning by attack channels, corruption patterns, or lifecycle stages, leaving formation interfaces insufficiently characterized. We formulate Memory Formation Poisoning as adversarial manipulation of the interfaces through which persistent agent memory is produced and committed, and introduce MemForm, a controlled diagnostic benchmark covering five interfaces: Content, Evidence, Scope, Selection, and Path Poisoning. We introduce Selection and Path Poisoning as controlled intervention categories that isolate retention and consolidation order effects, respectively. Each category pairs clean and poisoned conditions with interface-specific controls. Evaluations across six LLMs and two response formats reveal broad susceptibility at all five interfaces under the evaluated memory protocols and implementations. Crucially, authentic experiences with successful outcomes do not guarantee reliable behavior after memory formation: changes to contextual support, retention, or consolidation order can induce undesirable downstream behavior. In particular, consolidation order can alter behavior even with identical retained experiences, depending on the memory formation procedure. Additional analyses show that interface applicability depends on memory architecture and that Path Poisoning susceptibility depends on the updater. The evaluated defenses provide incomplete mitigation that varies by interface. These findings motivate protecting both what is committed to persistent memory and how memory is formed from experiences. Code and data are available at https://github.com/opanhw/memform-code.