Vision language models learn to forget unwanted visual styles across categories
Open Vocabulary Domain Unlearning
Computer Vision and Pattern RecognitionMachine Learning
Summary
Vision-language models can recognize many objects but sometimes remember unwanted visual styles, like cartoon cars or textbook diagrams, which can cause problems. The authors show that current methods to remove these styles only work for specific object types seen during training, not for new object types. They propose a new method that truly erases the unwanted style across all object categories, keeping the model accurate otherwise. Their approach finds and edits only the parts of the model related to the unwanted style, achieving better results with fewer examples.
What this means in practice
- •For medical ai developers: Remove idealized diagram styles from medical vision-language models to improve real-world clinical image analysis.
- •For autonomous vehicle engineers: Erase cartoon-style visual patterns from vehicle detection systems to enhance performance in realistic driving environments.
Authors
Sumanth Udupa, Mehrtash Harandi, Yadan Luo, Mahsa Baktashmotlagh
Abstract
Vision-Language Models (VLMs) exhibit remarkable zero-shot generalization, yet they often encode unwanted or hazardous stylistic domains such as idealized textbook diagrams in medical AI or cartoon vehicles in autonomous driving. Approximate Domain Unlearning (ADU) aims to selectively erase a model's recognition of a target visual domain while preserving accuracy on the remaining domains. However, existing ADU methods operate under a flawed closed-vocabulary assumption: they evaluate unlearning solely on the specific object classes seen during the unlearning fine-tuning phase. Consequently, these methods do not unlearn the domain itself; they merely overfit to seen class-domain pairs, leaving the domain easily recognizable for unseen classes and providing a false sense of removal. We argue that true domain erasure must be class-agnostic. To address this, we formalize Open-Vocabulary Domain Unlearning (OVDU), a rigorous protocol that mandates domain forgetting must transfer to held-out classes. To solve the OVDU challenge, we propose a surgical parameter-editing framework. First, a Fisher Information mask isolates domain-sensitive weights, mathematically protecting foundational zero-shot generalization. Second, our Targeted Manifold Scattering (TMS) objective uses preference-based mining to locally scatter the forget domain's stylistic geometry. Evaluated across PACS, OfficeHome, and DomainNet, our method vastly improves open-vocabulary generalization over existing baselines. Crucially, it delivers exceptional sample efficiency, outperforming peak 8-shot baseline results with only 4 shots.