AI summaryⓘ
The authors studied Vision-Language Models (VLMs) that sometimes give wrong answers about images, like calling a blue banana yellow. They found that the models actually recognize the right visual details early on but fail to use that information correctly when deciding on an answer. By analyzing the models layer by layer, the authors showed that problems happen not because of poor seeing, but because the model's later decision process favors prior knowledge over actual visual input. They also demonstrated ways to fix this by adjusting activations in early layers, improving accuracy without extra training. Overall, the research suggests VLMs see well but struggle to act on what they see, and targeted fixes can help.
Vision-Language ModelsVisual GroundingEncoding-Grounding DissociationMultimodal Arbitration CrossoverLogit Lens ProbingActivation PatchingVisual AttributesCausal InterventionActivation SteeringPrior Knowledge Bias
Authors
Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang, Jonathan Fürst, Kurt Stockinger
Abstract
When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding--Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit -- not the strength of encoding -- better predicts grounding outcomes with a correlation of . After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering -- both linear and sparse autoencoder-guided -- in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.