Vision language models forget unwanted domains by patch selective unlearning
Not All Patches Are Equally Forgettable: Spatially Localized Domain Unlearning in Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Pre-trained vision-language models are good at recognizing images across different environments without extra training. However, they can also keep unwanted information tied to specific domains, making it hard to remove just that part. The authors suggest looking at which small image patches contribute most to recognizing the unwanted domain and selectively weakening those, instead of changing the whole image. This approach helps the model forget unwanted domain details without hurting its main recognition abilities.
What this means in practice
- •For image recognition developers: Improve model robustness by selectively removing domain-specific biases in pre-trained vision-language models without harming overall image recognition.
- •For computer vision system integrators: Refine vision-language models to better handle varying environments by locally unlearning contributions from visual patches tied to unwanted domains.
Authors
Akanksha Singh, Vinod K. Kurmi
Abstract
Pre-trained vision-language models (VLMs) exhibit strong cross-domain recognition performance even without additional training. However, this robustness can also preserve undesirable domain-specific behavior, as domain-related and semantic information often remain entangled within the learned representation space, making selective domain unlearning challenging. Existing approaches typically address this problem through latent-space disentanglement and prompt- or feature-level interventions, without directly attributing and attenuating individual patch-token contributions. However, here we suggest that rather than uniformly suppressing the full representation, it may be more effective to exploit the spatial structure of vision transformers to localize and suppress patch regions that contribute disproportionately to forget-domain prediction. Patches that strongly influence forget-domain prediction may not be equally important for semantic recognition, suggesting that forgetting should be guided according to the domain contribution of different visual regions. Specifically, we propose a two-stage patch-selective framework that first estimates patch-level domain sensitivity and then selectively attenuates patches whose contribution to forget-domain prediction is stronger than their semantic utility. We evaluate our framework on Office-Home, Mini DomainNet, and DomainNet. Experimental results demonstrate improved forgetting-retention tradeoffs compared to prior methods while improving retained-domain recognition by up to 3.8\%. Additional evaluations under visually overlapping and unseen-domain settings further demonstrate improved robustness under distribution shift.