Backdoor attacks stopped in vision language models with selective channel fixes
Selective Channel Restoration for Backdoored Vision-Language Models
Cryptography and SecurityComputer Vision and Pattern Recognition
Summary
Vision-language models that understand both pictures and text can be tricked by sneaky backdoors that make them behave badly. The authors found that the part of these models that connects images and words is fragile when backdoored. They made a way called PSR that carefully fixes just the most sensitive parts after training, without slowing down the model later. This method stops the hidden tricks while keeping the model’s normal abilities intact.
What this means in practice
- •For machine learning engineers: Secure vision-language models during deployment by selectively repairing vulnerable channels to prevent backdoor exploits without extra inference cost.
- •For security operations teams: Integrate selective restoration techniques to reduce risks from poisoned training data in multimodal AI systems.
Authors
Shuming Liu, Zhifang Zhang, Suqin Yuan, Khin Mi Mi Aung, Zhuoyi Lin, Lei Feng
Abstract
Vision-language models (VLMs) exhibit strong multimodal capabilities but remain vulnerable to backdoors implanted through poisoned fine-tuning data. Existing defenses often require extensive parameter updates during fine-tuning or incur per-query overhead during inference. To address these limitations, we propose Perturb-Select-Restore (PSR), a post-training defense that performs sparse updates to the projection interface and introduces no additional computation during inference. We reveal that backdoored VLM projectors are substantially more sensitive to bounded perturbations than clean VLM projectors, a phenomenon we term projection fragility. Building on this finding, PSR identifies the output channels most sensitive to perturbations in each projection layer of a backdoored VLM and restores their parameters to the corresponding pretrained values. Experiments across multiple tasks show that PSR reduces attack success rates to near zero while preserving clean-task performance.