Vision language models choose like humans but look differently
Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Models
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
This paper looks at how humans and vision-language models (VLMs) make connections between made-up words and images, such as pairing 'bouba' with round shapes and 'kiki' with sharp ones. The researchers compared not only what choices both groups made but also where they looked while deciding. Although some bigger models made similar choices to humans, their visual attention did not match human eye movements. Fine-tuning models improved choice similarity but still did not align their 'gaze' with humans. This shows that making similar choices or looking patterns does not mean the models think like humans.
What this means in practice
- •For ai system developers: Improve vision-language model training by incorporating human choice data to match human-like decision patterns on unseen stimuli.
- •For user interface designers: Design interfaces that better predict or respond to where users look by understanding that model attention does not always align with human gaze.
Authors
Sumin Hong, Katsumi Ibaraki, Renee Shi, David Chiang, Toby Jia-Jun Li
Abstract
Cross-modal associations are systematic pairings of features across modalities, such as the association of 'bouba' with round shapes and 'kiki' with sharp shapes. Prior work has compared humans and vision-language models (VLMs) on such associations, but often using different stimuli or tasks between humans and models. Here, we ask whether VLMs align with humans not only in choices, but also in where they look when making those choices. We study both VLMs and humans (N = 53), presenting them with the same stimuli, a pseudo-word and two images, and record participants' choices and eye movements, which we release. We find choice alignment in a few larger VLMs, but their saliency matches human gaze less closely than a center-bias baseline, a fixed Gaussian at the center of each image. Fine-tuning small VLMs on human choices brings their choice alignment to the level of a human majority-vote reference on unseen words and images, yet their attention still matches human gaze less closely than this baseline. Training model attention on human gaze raises attention-gaze correlation without improving choice alignment, and a single average gaze map per image position raises it by a similar amount. Matching human choices, or even human gaze patterns, is therefore not sufficient evidence of human-aligned cross-modal processing.