Language models forget facts by cutting reasoning links
Making LLMs Truly Forget: Deep Unlearning by Searching, Selecting, and Severing Knowledge Paths
Machine Learning
Summary
Sometimes when you teach a language computer model to forget a fact, it can still guess that fact by connecting related ideas. The authors show that to really erase a fact, you have to break the links the model uses to reason it out. They create a way to find all the reasoning paths inside the model and then cut them so the fact cannot be recovered. Their method makes the model forget more deeply without losing other knowledge.
What this means in practice
- •For machine learning engineers: Remove sensitive or outdated facts from language models by cutting supporting reasoning paths to ensure deeper knowledge erasure.
- •For data privacy teams: Ensure compliance with data removal requests by thoroughly eliminating recoverable facts rather than just isolated ones in AI models.
Authors
Jialu Wang, Peizhi Niu, Haoteng Yin, Hans Hao-Hsun Hsu, Pan Li, Rongzhe Wei
Abstract
While an unlearned language model may no longer recall a fact directly, the fact often remains recoverable through multi-hop reasoning over related knowledge. Most existing unlearning techniques overlook this vulnerability, targeting facts in isolation while leaving their supporting knowledge intact. To achieve true forgetting, we propose a general deep unlearning framework compatible with existing unlearning algorithms. Our approach adaptively explores both explicit responses and latent internal representations to discover valid reasoning paths, compiles them into a confidence-aware supporting subgraph, and we apply a graph minimum cut to sever all recovery paths while preserving unrelated knowledge. To rigorously evaluate deep unlearning, we introduce a model-specific pipeline that extracts and completes knowledge graphs from raw text, filtering them by calibrated model confidence to reflect what the model genuinely retains. Comprehensive experiments demonstrate that selectively unlearning supporting knowledge yields substantially deeper forgetting than superficial methods while preserving model utility, highlighting that genuine unlearning requires breaking the relational structures that enable factual reconstruction.