Selective forgetting method improves privacy in vision language models
SIEVE: Selective attention-value Suppression for Vision-Language Models Unlearning
Machine LearningComputer Vision and Pattern Recognition
Summary
Vision-language models can connect images of people with personal information, which raises privacy concerns. The authors propose a new method called SIEVE that helps these models 'forget' sensitive personal data while keeping other information intact. This is tricky because the same pictures can contain both private and allowed information. SIEVE works by adjusting the model’s attention patterns during learning to suppress forgotten info but keep what should remain. Tests show that SIEVE effectively removes targeted personal info without hurting the model's overall knowledge too much.
What this means in practice
- •For ai privacy engineers: Implement targeted data removal in vision-language systems to comply with privacy regulations without harming useful model capabilities.
- •For multimodal system developers: Improve control over what visual and textual knowledge remains in complex AI models by selectively suppressing attention signals.
Authors
Si Qi Goh, Cap Dang Xuan Kiet, Tat-Jen Cham, Kwok-Yan Lam
Abstract
The ability of vision-language models (VLMs) to associate visual identities with biographical information creates a need for selective unlearning of personally identifiable information (PII) while preserving permitted knowledge about the same individual. This setting is challenging because both sensitive and retained information can share the same visual inputs and intermediate representations. We introduce SIEVE, a simple and effective framework for selective VLM unlearning. SIEVE directly regularizes attention-value representations while also controlling model outputs. SIEVE suppresses attention values for forget examples toward a constant zero, while preserving retain-example representations by matching them to a frozen reference model. These objectives are combined with sequence-level forget and retain supervision, enabling targeted forgetting without largely affecting retained knowledge. Extensive experiments show that SIEVE achieves state-of-the-art performance on unlearning with multiple model-modality settings, while maintaining competitive retained utility. Ablation studies further show that value suppression and negative cross-entropy contribute complementary forgetting signals, while reference-based value matching substantially reduces utility degradation. These results demonstrate that attention values provide an effective intervention point for selective multimodal unlearning when sensitive and retained knowledge are closely related.