Persistent memory in language agents increases privacy leak risks over time
Disclosure Begets Disclosure: A Self-Reinforcing Privacy Loop in LLM Agents with Persistent Memory
Cryptography and SecurityComputation and LanguageMachine Learning
Summary
When AI language agents remember personal details over many conversations to improve answers, this memory can cause privacy problems. The authors found that the more an agent recalls and mentions user details, the more users end up sharing or confirming these details, making the information easier to guess later. This creates a feedback loop where the agent’s guesses get recorded and repeated, increasing privacy risks. The problem can be reduced by instructing agents not to mention or assume personal details, keeping quality high but limiting data leaks.
What this means in practice
- •For customer support teams: Design support chatbots to avoid restating user personal details to reduce privacy leaks while maintaining high response quality.
- •For healthcare chatbot developers: Implement guidelines that instruct healthcare AI agents not to assume patient details to lessen disclosure of sensitive data over multiple sessions.
Authors
Masahiro Kaneko, Hiromi Arai, Timothy Baldwin
Abstract
Large language model (LLM) agents increasingly use persistent memory to personalize responses, and personal attributes in this memory pose a privacy risk if exposed. We ask how consulting memory during dialogue changes what users go on to say, and thus what the memory comes to reveal. In multi-session simulations, a simulated user with 24 personal attributes talks with a support agent that does or does not consult memory, and in both settings attributes are inferred from the full accumulated record. Across four support models, memory raises response quality but also empirical leakage, the share of attribute values correctly inferred from the record, by a margin that widens over sessions. The cause is a loop in ordinary, non-adversarial assistance. Memory-conditioned responses carry earlier values forward, often as presuppositions, and users affirm them or leave them uncorrected, so that cues and the agent's own guesses become recorded values. The loop reinforces itself, and removing the agent's mentions of an attribute lowers its later disclosure about twice as much as single-turn effects predict. Users cannot leave it on their own, since correcting a value discloses the attribute, declining to discuss it shifts later predictions toward sensitive values, and incorrect values left in place become entrenched. The loop is best weakened at the agent's response. Instructing the agent not to state or presuppose attributes brings leakage to about the no-memory level while keeping most of the quality gains, whereas filtering or anonymizing what users say or what is stored gives up much of them.