Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Models
2026-08-31 • Computation and Language
Computation and LanguageComputer Vision and Pattern Recognition
AI summaryⓘ
The authors study how to remove one person's data from large language models that also process images, without needing extra data that is hard to get after the model is ready. They find that information about that person is mostly stored in certain parts of the model called early-to-mid decoder MLPs. Their method, called PAVA, carefully updates only those parts while using a special technique to keep the model's ability to understand images intact. Tests show that PAVA does a good job of forgetting the person’s data without hurting the model's overall performance compared to other methods.
multimodal large language models (MLLMs)unlearningforget setretain setdecoder MLPscausal tracingweight transplantFisher overlapknowledge distillationimage-grounded behavior
Authors
Kangwook Ko, Jaehyuk Jang, Wonjun Lee, Hee-Seon Kim, Changick Kim
Abstract
Removing a specific individual's information from multimodal large language models (MLLMs) is often needed after deployment, but existing methods rely on a retain set, which is hardest to obtain at that point, and rebuilding it recreates the privacy exposure that unlearning aims to remove. Forgetting from the forget set alone instead damages the shared visual-language computation, harming perception. We cast retain-free unlearning as a localization problem: causal tracing, weight transplant, and Fisher overlap all point to early-to-mid decoder MLPs as the layers where identity information is stored and, unlike other module families, can be modified without substantially disrupting vision. We turn this into Pathway-Aware Visual-attribute Anchoring (PAVA), which confines updates to these layers and pairs a forget loss with a visual-attribute anchor that preserves image-grounded behavior by distilling the model's own pre-unlearning answers from the forget images alone. On MLLMU-Bench and ReMem, PAVA gives the strongest forget-retain trade-off among forget-set-only methods and remains competitive with retain-based baselines.