Vision language models rely mainly on image patch position codes for left right
Where Do Vision-Language Models Get Left and Right? Position-Code Counterfactuals in Qwen2.5-VL, Qwen3-VL and InternVL3
Computer Vision and Pattern RecognitionMachine Learning
Summary
Figuring out left and right in images is tricky for AI models that understand pictures and words together. The researchers found that these models mostly get the left-right direction from special codes tied to where parts of the image are, rather than from the order of words or language tokens. By flipping just the image position codes, the models changed their left-right answers almost as if the image itself was flipped. This understanding helps show which parts of the AI models are important for recognizing directions in images.
What this means in practice
- •For computer vision engineers: Improve image and text fusion by focusing on position codes to better handle spatial direction questions in vision-language systems.
- •For natural language processing developers: Enhance multimodal AI models by understanding that token order and language model mirroring have limited effect on spatial direction interpretation.
Authors
Enhao Cui, Junzhi Yang, Yaoshun Lin, Yuhan Feng, Zijian Chen
Abstract
A vision-language model may take left-right direction from the orientation of image content within patches, its vision transformer's (ViT) position code, or token positions and order in its language model (LLM). Ablations show the ViT code is necessary; we test sufficiency by mirroring only a position code. In Qwen2.5-VL-7B, Qwen3-VL-4B and InternVL3-2B, mirroring ViT width positions reproduces the flipped-image answer on 84.3-93.2% of CV-Bench left-right questions (Qwen2.5-VL-3B: 87.5%, partly a left fallback). With two options, this also counts disruption; a half-grid shift flips most answers too. With the ViT code intact, LLM-side id and token-order mirrors change few left-right answers in all three series (a weak id test in Qwen2.5-VL). In Qwen2.5-VL-7B the effect sits in the global-attention ViT blocks, mostly the last. On What'sUp, flipping content and token order, not the position code, keeps more on/under than left-right answers at the flipped-image answer. Post hoc analyses qualify these results. On CV-Bench items with located objects, random permutations flip 18.9-31.0 points more answers when reversing the objects' order than when keeping it. On What'sUp, in Qwen2.5-VL and Qwen3-VL, the share of the left-right changes made by random permutations that land on the mirror relation is mostly below chance. Counted as returns to the original answer, the What'sUp contrast of support with left-right relations is clear only in InternVL3-2B. Most remaining errors give the same relation whichever object is asked about, suggesting that repair may target relation binding, object localization or use of the reference object.