Summary
When information is deleted from a language model agent's memory, traces of that information often remain in related summaries or other derived data. The authors created a test called AfterDeleteBench to check if deleted secrets can still be recovered from the agent’s behavior. They found that many large language models can reconstruct deleted secrets just by querying them, even if the exact secret is not stored anywhere anymore. This shows that simply deleting the original data does not guarantee the information is really gone from the system’s memory. The authors suggest that deletion should be checked by testing how the agent behaves, not just by verifying the database is cleared.
What this means in practice
- •For ai developers: Verify that deleting data from agent memories effectively removes all traces by testing agent behavior, preventing unexpected data recovery.
- •For data privacy teams: Audit AI systems for compliance failures by detecting information leakage after deletion requests through behavioral tests like AfterDeleteBench.
Authors
Ehsan Saleh, Roozbeh Nahavandi, Sama Salarian, Raef Bassily, Andrew Perrault
Abstract
Many LLM agents maintain external persistent memory and derive additional state from past interactions, such as summaries. Deleting an original memory record therefore need not remove information carried by its descendants. We introduce AfterDeleteBench, a controlled behavioral audit for this failure mode in fixed-weight LLM agents. For each synthetic secret, we compare an exposed-then-deleted agent with an otherwise matched agent that was never exposed to the secret, allowing post-deletion recovery to be attributed to the original exposure. In our primary setting, a single query recovers the deleted secret from Qwen2.5-7B-Instruct in roughly two thirds of cases, while matched never-exposed controls have zero recovery. Crucially, recovery does not require a surviving verbatim copy: when the complete secret string is absent and only separately stored components remain under a public composition rule, exact reconstruction succeeds in 62-99% of cases across Qwen, Mistral, and Phi-3. We also show that the response interface is part of the measurement system: target-bearing replies can be scored as failures when the requested format and parser are misaligned. Finally, we separate reconstruction from write-back and fresh-session recall. Provenance-aware closure eliminates observed recovery in the tested positive-baseline settings, whereas an exact-literal write filter blocks reinsertion but cannot erase information already present. Together, these results show that source deletion should be evaluated behaviorally at the level of the agent and its memory system, not only as a storage operation.