Llm memories can be erased correctly using a new deletion system
Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memories
Computation and Language
Summary
Large language models (LLMs) can remember past conversations using external storage, but currently they cannot truly forget information when asked. The authors show that simply deleting messages is not enough because conversations connect messages in complicated ways, making information stick around. They introduce DeLLM, a system that keeps track of all messages and their connections to properly erase memories. Their tests show that DeLLM can effectively forget requested information while still working well.
What this means in practice
- •For enterprise chatbot developers: Enable chatbots to permanently erase customer data on request while maintaining conversation quality.$Commercial implications: Provides a product feature for privacy compliance in enterprise chatbots, differentiating services in regulated markets.
- •For digital assistant builders: Create assistants that can selectively forget user-shared information, enhancing control over personal data.
Authors
Olga Ohrimenko
Abstract
We consider persistent LLMs that accumulate memories of their interactions with a user over time. Such LLMs maintain memories using external storage, which they can query to overcome the limitations of a fixed context window. Such systems have numerous practical applications, as they can draw on all past interactions when responding to user queries. In this paper, we ask whether LLMs can forget information shared with them upon a user's request. We find that current LLMs fail to delete such information---even when they claim to have forgotten it and even when operating with a limited context. To this end, we consider a new direction of study: Deletion of LLM Memories. We show that naively removing messages that match a user's deletion request is insufficient, since conversations naturally introduce message dependencies that cause information to persist. To correctly handle deletion requests, we propose the DeLLM framework. It dynamically constructs relevant context for each LLM query and maintains a provenance graph of messages to determine which ones must be removed during deletion. Our experiments show that DeLLM achieves a high deletion rate while maintaining utility.