Multimodal models improve visual reasoning by focusing selective attention
Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Multimodal language models sometimes answer questions without really looking at the important parts of images, which can cause mistakes. The authors found a way to help these models pay attention only to the key visual details by guiding the parts of the model most sensitive to images. This selective attention improves how well the model uses visual clues to reason and answer questions. Their method showed better results across many tests, proving that focusing the model’s attention on the right image areas helps it think better.
What this means in practice
- •For multimodal ai developers: Enhance visual question answering systems by guiding attention to relevant image parts to reduce hallucinations and improve accuracy.
- •For computer vision engineers: Design vision-language models that better ground their reasoning on meaningful spatial regions within images, improving interpretability and reliability.
Authors
Jiaqi Deng, Zonghan Wu, Zhan Heng, Xiaoshui Huang, Huan Huo, Guandong Xu
Abstract
Multimodal large language models (MLLMs) achieve strong performance on visual reasoning tasks, yet remain prone to hallucinations and over-reliance on language priors, often generating answers without adequately using task-relevant visual evidence. Existing approaches primarily improve reasoning through reasoning-oriented supervision or inference-time strategies. In this work, we study a complementary question: can multimodal reasoning be improved by strengthening implicit visual grounding without directly supervising the reasoning process? Motivated by the functional specialization of attention heads, we investigate whether reasoning can be improved by guiding only the heads most responsive to visual evidence grounding. We propose Selective Probability Mass Concentration (sPMC), a training framework that identifies grounding-responsive heads and selectively regularizes their text-to-image attention. sPMC treats normalized attention over visual tokens as a spatial probability distribution and encourages the probability mass to be assigned to semantically relevant regions using segmentation-derived spatial priors. Adaptive Head Selection restricts this guidance to visually responsive heads while leaving the remaining heads unconstrained to preserve their complementary functions. Across 6 multimodal benchmark suites, sPMC achieves an average zero-shot improvement of 3% and gains of up to 11.3% across multiple MLLMs while regularizing only 3%-15% of their attention heads. These results demonstrate that targeted guidance of sparse and implicit visual evidence pathways can directly improve multimodal reasoning.