Modality order affects judgments in multimodal AI conflicts
Same evidence, different judgments: Evidence noncommutative in vision/speech-text conflicts
Artificial Intelligence
Summary
Sometimes AI models get clues from both pictures or speech and text, but what happens if these clues don’t agree? The authors found that the order in which AI models see these clues matters a lot. If the picture or speech comes after the text, the model is more likely to believe the picture or speech. This means that AI answers can change just by swapping the order of the same information.
What this means in practice
- •For multimodal ai developers: Design AI systems that manage conflicting text, image, or speech inputs by adjusting evidence order to balance modality influence on final decisions.
- •For user interface designers: Improve how multimodal interfaces present information to guide AI responses by controlling the sequence of displayed or spoken data.
Authors
Zhuoyun Li, Boxuan Wang, Xiaowei Huang, Yi Dong
Abstract
For multimodal large language models, when images or speech conflict with accompanying text, measured text reliance can entangle modality preference with evidence position. Earlier studies of text bias often used a fixed evidence order or moved task instructions with the evidence, leaving the contribution of order unclear. In this paper, we use a paired comparison that keeps the instructions and evidence content fixed and swaps only the positions of the two sources to quantify this potential influence. Across vision and speech models, placing an image or recording after conflicting text consistently shifts answers toward its content. We also revisit previous studies and analyze why their experimental settings can lead to misleading conclusions. These findings reveal cross-modal evidence noncommutativity: the same evidence can lead to different judgments when its order changes, and placing perceptual evidence later can increase the model's reliance on its content.