EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensation
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
The authors found that large vision-language models often lose important visual details as they generate answers, making some responses unsupported by the images. To fix this, they created EviAnchor, a method that keeps visual information alive and easy to access during the entire answer process without extra training. EviAnchor uses special slots to gather and organize visual details and helps the model focus on these details rather than just the text. Their tests show this approach improves how well the model connects answers to the images.
Authors
Sihang Jia, Shuliang Liu, Songbo Yang, Xuming Hu
Abstract
Large vision-language models (LVLMs) frequently generate content unsupported by visual inputs. Preliminary experiments show that visual evidence is primarily incorporated into answer-side representations in early-to-middle decoder layers, while its direct influence progressively weakens in later layers. This attenuation suggests that visual evidence acquired earlier may be insufficiently utilized during subsequent generation. Based on this observation, we propose EviAnchor, a training-free and single-branch inference framework that preserves and reactivates visual evidence throughout generation. EviAnchor introduces Regional Evidence Anchor (REA) slots to progressively aggregate dense visual tokens into spatially structured representations. It then strengthens the current decision state's access to these visual anchors through decision-conditioned evidence routing, mitigating excessive dependence on textual context. Finally, the model resumes its native Transformer computation to integrate the retrieved visual evidence with question semantics and generation history. Experiments across POPE, CHAIR, and MMHal-Bench demonstrate consistent improvements in visual grounding.