AIM: Anchor Identity Features, Then Match for Multimodal Large Language Model Unlearning
Computer Vision and Pattern RecognitionComputation and Language
Summary
The authors looked at how large AI models that handle both images and text can accidentally remember personal details about people, which is a privacy risk. They focused on how to make the model forget specific people's identities when some data (images to keep) isn't available during deletion. They found that the model's understanding of identity and general image recognition happens in different parts of its memory. Using this, the authors created a method called AIM that helps the model forget a person’s identity without losing its ability to recognize or understand images in general. Their tests showed AIM works well at removing identity info while keeping other knowledge intact.
Authors
Wonjun Lee, Jaehyuk Jang, Kangwook Ko, Hee-Seon Kim, Changick Kim
Abstract
Multimodal large language models (MLLMs) can memorize identity-specific facts about people in their fine-tuning data, creating privacy risks when a person requests deletion. Existing MLLM unlearning methods often assume access to retain images or ground-truth answers during deletion, which is unrealistic in many practical scenarios. We study identity unlearning when retain images are unavailable at deletion time. Our analysis shows that identity and visual-perception questions occupy distinct regions in fine-tuned hidden states and are organized differently: identity questions cluster by person, whereas perception questions cluster by question type. This suggests that identity knowledge can be suppressed without erasing general visual perception. Building on this observation, we propose AIM, a two-stage method that anchors an identity-forgetting target with a universal visual prompt and then matches the vision encoder to that target under a Fisher-based constraint. Extensive experiments show that AIM achieves competitive identity forgetting while preserving non-deleted identities, prior knowledge, and visual perception on the same images.