LLMs can hide sensitive data despite appearing to forget it
Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it
Machine Learning
Summary
Sometimes big language models seem to forget sensitive information when they no longer show it in their answers. This paper shows that even if the model's replies hide this info, the data can still be found inside how the model thinks at each step. The authors tested several ways to find this hidden info inside the model and found it often sticks around unnoticed. To fix this, they created a new method that actively removes hidden traces, giving stronger proof that the model truly forgets unwanted information.
What this means in practice
- •For ai safety teams: Ensure removal of sensitive data from models by verifying and suppressing hidden internal memory beyond just output checks.
- •For legal compliance officers: Achieve stronger guarantees that AI models have fully forgotten data required to be deleted under privacy regulations.
Authors
Hadi Reisizadeh, Jiajun Ruan, Sijia Liu, Mingyi Hong
Abstract
Unlearning in large language models (LLMs) is typically evaluated at the output level, where a model appears to suppress sensitive or undesirable content. In this work, we show that such evaluations can create an illusion of forgetting: even when output-level leakage is eliminated, sensitive information can remain encoded in the model's hidden representations. We first provide a theoretical analysis establishing a fundamental separation between output suppression and representational erasure. Specifically, we show that the decoder can be made arbitrarily insensitive to sensitive directions, driving output-level leakage to zero, while the hidden representations retain the underlying information. To empirically validate this phenomenon, we train generative probe decoders on hidden states across transformer layers, enabling layer-wise measurement of information leakage. Across three widely used benchmarks, TOFU, MUSE, and WMDP, and state-of-the-art unlearning methods, we find that substantial sensitive information remains recoverable from hidden representations, even when standard output-level metrics indicate successful unlearning. To address this gap, we propose Probe-Adversarial Representation Suppression (PARS), an unlearning objective that adversarially minimizes the extractable information from hidden representations. PARS directly targets representational leakage and provides significantly stronger guarantees of erasure under adversarial probing and relearning attacks, outperforming all evaluated baselines. Our results highlight a fundamental limitation of existing unlearning paradigms and suggest that true forgetting in LLMs requires controlling not only model outputs, but also the information encoded in hidden representations. Codes are available at https://github.com/OptimAI-Lab/HiddenStateUnlearning.