Vision language models weigh conflicting text and images with bias
Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
When vision-language models see images and read text, they sometimes get conflicting information. The authors explain this by comparing the models to biased Bayesian observers that combine clues from both sources but with their own preferences. Their two-parameter model can predict how these models choose between images and text even when the quality of one changes. This helps clarify why sometimes the model trusts text more and other times it relies on visual cues.
What this means in practice
- •For ai system designers: Improve multimodal AI by tuning Bayesian integration parameters to better balance text and image input reliability.
- •For product managers for ai assistants: Design more reliable AI assistants that respond appropriately when text and image inputs contradict by adjusting model biases.
Authors
Josh Barua, Kyle Mahowald, William Rudman
Abstract
Vision-language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear. We develop a predictive account from a Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases. The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting. Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and vision dominance under a single combination rule.