Adaptive steering reduces hallucinations in large vision language models
Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding what an image really shows can be tricky for computer models that read pictures and text together, and they sometimes imagine things that aren't there, called hallucinations. The authors studied how these models mix information from pictures and text while deciding what to say. They discovered that the models naturally pay attention to images in a way that can be guided to avoid mistakes. Using this insight, they made a method called AIMS that smartly balances attention to visuals and text without extra training. This approach helps the models avoid imagining incorrect objects while still performing well on many tasks.
What this means in practice
- •For multimodal ai developers: Improve image-text generation systems by reducing false object mentions without retraining the model.
- •For software engineers in content generation: Incorporate adaptive steering to enhance reliability of captioning and visual question answering services.
Authors
Shuran Ma, JiaLe Li, Yuxin Dong, Shan Zheng, Qingyun Jiang, Xiang Chen, Qi Zhu, Deyi Ji, Yifan Yang, Jianfeng Pan, Yu Tian, Xue Yang
Abstract
Hallucination remains a significant challenge in Large Vision-Language Models (LVLMs). Existing training-free methods generally mitigate hallucinations through contrastive decoding or visual enhancement, often increasing the relative influence of visual evidence during generation. This raises a fundamental question: Can LVLMs dynamically regulate the contributions of different context sources to suppress hallucinations? In this work, we investigate and quantify how LVLMs coordinate multiple context sources during decoding and examine how this intrinsic behavior can guide hallucination mitigation. We find that LVLMs exhibit an intrinsic vision-attending tendency that can guide adaptive visual steering, while textual contexts can also contribute to hallucination mitigation. Motivated by these findings, we propose AIMS (Adaptive Information Multi-source Steering), a lightweight training-free framework that adaptively coordinates visual, prefilled textual, and generated contexts during decoding. Specifically, AIMS constructs compact prototypes for the three context domains and estimates their affinities with the current query to determine head-wise steering weights. The resulting multi-source steering direction is applied to the query representation, enabling adaptive context integration without additional model training or auxiliary forward passes. Extensive experiments across multiple LVLMs and decoding strategies demonstrate that AIMS effectively mitigates object hallucination while maintaining competitive general-purpose multimodal capabilities.