Vision language models prefer text over color when features conflict
Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm
Computer Vision and Pattern Recognition
Summary
When vision-language models answer questions about images that contain conflicting clues, like the word 'RED' written in blue ink, they tend to focus more on the text than the color. The authors used a classic psychology test called the Stroop task to show this bias. They also looked inside the models and found that text features are more strongly and clearly represented than color features. This helps explain why it's easier to guide models using text information than color information. Overall, the study reveals that vision-language models treat different parts of an image with unequal importance when deciding how to respond.
What this means in practice
- •For ai system developers: Design better guidance methods by focusing on text features to influence vision-language model responses more effectively in multimodal tasks.
- •For user interface designers: Create image-based question answering systems that account for feature saliency biases to improve interface clarity and user trust.
Authors
Nurbüke Teker, Rui Xiao, Zeynep Akata, Shuchen Wu
Abstract
When you ask a vision-language model (VLM) a question with an image, it answers your question based on the image. But to act means to choose: saying one thing means choosing to respond not say another. This becomes an issue when the image contains multiple features that equally relate to the question at hand: how would VLMs choose to respond? This problem is precisely studied by the classical Stroop task in psychology: among a set of deliberately designed images whose text and color conflicts (e.g., \textcolor{blue}{RED}), participants are instructed to report the color. We adapt this psychological paradigm to a benchmark and prompted a range of contrastive and generative VLMs on their behavior. Across models, we observe a persistent bias toward the feature of text in Stroop images: when text and color disagree, models preferentially generate responses centered to the text rather than color. We then opened the black box and examined the source of this bias and why certain features influences the model's behavior more than others. In the embedding space, we found that textual cues are represented more strongly and separably than color cues. This representational asymmetry predicts intervention success: steering models toward text-related directions is substantially easier than steering them toward color-related directions. Using a classical psychological paradigm, we demonstrate, at both behavioral and representational levels, that distinct features within the same image can exert unequal influence on VLM behavior. Our findings suggest that VLMs assign different degrees of saliency to different features within the same image, highlighting the need to understand how cue-specific saliency shapes their performance in widely used downstream applications.