Visual references improve multimodal reasoning by closing reference gap
Diagnosing the Reference Gap: What Visual Scaffolds Can and Cannot Fix in Multimodal Reasoning
Computer Vision and Pattern RecognitionComputation and Language
Summary
Multimodal language models can understand pictures but often struggle to think through visual information step-by-step, like keeping track when counting objects. The authors found this is partly because the models fail to clearly link language to specific parts of the image, a problem they call the reference gap. They created a test with visual tasks that keep the picture and question the same but change how references are shown, like adding numbers to objects. Their results show that giving the models clear, matching visual cues improves their reasoning, while mismatched cues don’t help or even make things worse.
What this means in practice
- •For multimodal model developers: Improve visual reasoning accuracy by adding task-matched visual references without extra training.
- •For human computer interaction designers: Design user interfaces that provide clear visual cues aligned to tasks to support AI-assisted image understanding.
Authors
Guizhen Chen, Weiwen Xu, Wotao Yin, Anh Tuan Luu, Hou Pong Chan
Abstract
Multimodal large language models can recognize visual content yet struggle to reason reliably over it across multiple steps. In counting, a model may see all objects but lose track of which ones it has counted. We argue that such failures, commonly attributed to perception, reveal a reference gap: the failure to bind language to the relevant visual entities, regions, or locations. Motivated by the theory of visual routines, we test whether external references such as numbering can narrow this gap. We introduce ReGap-Bench, a paired benchmark of five visual reasoning tasks in which the scene, question, and answer are fixed and only the reference varies. Results show that task-matched references improve accuracy by up to 25\%, while mismatched references provide little benefit or even hurt performance. These findings identify the reference gap as a bottleneck distinct from visual perception and show that constructing task-appropriate visual references can serve as a training-free visual skill for improving multimodal reasoning.