Disentangling Semantic Attention from Structural Bias in the Attention Manifold
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors found that Multimodal Large Language Models (MLLMs) often pay too much attention to certain unhelpful parts of images, which they call visual attention sinks. Instead of fixing each distracting part separately, they suggest this problem is due to a broader bias that makes the model trust text clues more than actual visual details. To fix this, they developed SPAR, a method that cleans up this bias and redirects attention to important image areas without needing extra training. Tests show SPAR helps these models understand images better with little extra computing cost.
Multimodal Large Language ModelsAttention MechanismVisual Attention SinksTextual BiasMultimodal HallucinationsVisual GroundingSaliency-guided PurificationAdaptive RedistributionPlug-and-play InterventionComputational Efficiency
Authors
Pengkun Jiao, Bin Zhu, Jingjing Chen, Yu-gang Jiang
Abstract
The empirical success of attention mechanism in Multimodal Large Language Models (MLLMs) often obscures its inherent, subtle flaws. Specifically, MLLMs consistently exhibit disproportionate attention toward certain semantically uninformative visual tokens, a phenomenon termed "register" or "Visual Attention Sinks." While existing inference intervention methods attempt to identify these sink tokens and redistribute their attention weights, such approaches typically treat these tokens in isolation and suffer from computational inefficiency. Instead, we reframe this phenomenon as a generalized textual bias exerted over visual features that extends beyond isolated sink tokens. From this perspective, a pervasive structural bias leads to the dilution of the semantic visual signal, precipitating multimodal hallucinations as the model prioritizes linguistic priors over valid visual evidence. To address this limitation, we introduce Saliency-guided Purification and Adaptive Redistribution (SPAR), a training-free, plug-and-play intervention. SPAR mitigates this generalized textual bias by purifying structural noise and subsequently redistributing the reclaimed attention budget to the most informative visual regions. Comprehensive evaluations across a diverse spectrum of hallucination benchmarks demonstrate that SPAR effectively restores authentic visual grounding with negligible computational overhead.