Vision language models reduce hallucinations using spectral attention shaping
Beyond Attention Imbalance: Mitigating Hallucinations via Spectral Surgery
Computer Vision and Pattern Recognition
Summary
Models that combine vision and language sometimes make mistakes called hallucinations, where they describe things that aren’t really there. The authors found that problems happen not just because of how the model pays attention to images and words, but because of certain hidden patterns in the way visual information flows inside the model. To fix this, they created a new way called FLASH that adjusts these hidden patterns without making the model slower or needing extra training. Their tests show that FLASH helps reduce errors while keeping the model efficient.
What this means in practice
- •For ai development teams: Improve the accuracy of vision-language AI systems by reducing hallucinations without increasing inference time significantly.
- •For cloud ai service providers: Integrate spectral attention shaping to offer more reliable and efficient vision-language model APIs.$Commercial implications: Enables selling improved vision-language AI services with lower error rates and faster response times.
Authors
Siqi Lu, Suo Wei, Yongbin Zheng, Jianhang Yao, Wanying Xu, Peng Wang
Abstract
While Large Vision-Language Models (LVLMs) achieve remarkable success, hallucinations remain a significant barrier to their reliable deployment. Recent studies primarily attribute these issues to cross-modal attention imbalances; most solutions therefore focus on reweighting visual tokens or suppressing language priors. However, such approaches often overlook the spectral characteristics of the visual information flow and frequently rely on Contrastive Decoding (CD), which doubles inference time. Instead of following conventional approaches, we identify two distinct hallucination patterns-Perceptual-Semantic Dissociation and Localized Fixation-and propose FLASH (Frequency-Localized Attention SHaping), a training-free and CD-free framework. FLASH utilizes a Spectral Vortex Score to detect vision heads within multi-head attention layers and applies adaptive spectral modulation to rectify the visual information flow during decoding. Empirical results demonstrate that FLASH achieves a superior balance between performance and efficiency compared to SOTA methods.