Language decoder governs vision-language model adversarial robustness
Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Some vision-language models can be tricked during training to accept wrong captions, but still produce correct descriptions when generating text normally. The authors studied why this happens in a large model called Qwen2.5-VL-7B-Instruct. They found that the problem isn’t mainly with the image understanding part, but with how the language part processes and decides which words to output. This means defenses against such attacks should focus more on the language model side than on the visual encoder.
What this means in practice
- •For ai system developers: Improve robustness defenses in vision-language models by focusing on language decoder components rather than just visual encoders.
- •For model evaluation teams: Design faithfulness evaluations that target the language decoder’s prior to detect adversarial vulnerabilities more effectively.
Authors
Arun Josephraj Arokiaraj, Zekun Wu, Adriano Koshiyama
Abstract
A targeted adversarial perturbation can drive a vision-language model's (VLM's) teacher-forced training loss for a fixed target caption to near zero, yet the same model, allowed to generate freely, produces the original, correct description with no trace of the target. We call this dissociation the train/inference gap, and give it a precise mechanistic account on Qwen2.5-VL-7B-Instruct using a controlled two-stage PGD attack on 200 held-out COCO images. First, we show that image-level pixel statistics, including a correctly re-implemented, texture-based attackability measure from the CNN robustness literature, have essentially no predictive power over which images are corrupted (best predictor r=-0.050, p=0.484; ridge regression R^2=0.069). Second, using the logit lens, we localise the gap to a single autoregressive step: the rank of the target token, conditioned on the correct first token already being generated, is fixed at exactly 3,488 out of 152,064 vocabulary entries for every image and every condition, with zero variance. Third, tracking target-token rank across all 28 LLM decoder layers reveals that the visual encoder corrupts every image's representation by a comparable margin regardless of eventual outcome, but the language model decoder then differentially arbitrates: amplifying the corrupted signal for susceptible images and actively suppressing it, past its clean-image baseline, for resistant ones (p<0.001, rank-biserial r=0.579). A linear probe on the merger hidden state separates these two outcomes with AUC=0.858, though we flag a circularity concern in this estimate. Together these results argue that adversarial robustness in autoregressive VLMs is substantially a property of the language decoder's prior, not the visual encoder, with direct implications for where faithfulness evaluations and defenses for deployed VLM systems should be targeted.