Visual language models improve image understanding using guided attention
SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders
Computer Vision and Pattern RecognitionComputation and Language
Summary
Large AI models that combine pictures and words sometimes struggle to focus on the right parts of an image, which can cause mistakes. The authors found that some layers in these models always look at the same few areas, regardless of the question asked. They created a technique called SAGE that helps the model pay attention to the right image parts based on the question, using existing tools that find important regions in images. This approach helps the models understand images better and makes their answers more accurate, especially when details matter.
What this means in practice
- •For image captioning developers: Improve captioning systems by guiding models to focus on relevant image parts, reducing mistakes caused by incorrect attention.
- •For robotics perception teams: Enhance robots' visual understanding by steering AI attention to important regions, improving object recognition and task performance.
Authors
Jeonghyo Song, YoungJoon Yoo
Abstract
Recent large vision-language models (VLMs) pair a visual encoder with a large language model (LLM) and perform well on diverse image-text tasks, yet their reliability is often limited by decoder attention pathologies that suppress visual evidence and exacerbate hallucinations. In this paper, we revisit visual attention sinks and uncover a structured, layer-dependent behavior: across prompts, early and late decoder layers exhibit prompt-invariant attention collapse onto the same few image regions, which we term PIS (Prompt-Invariant Sinks), whereas mid layers become prompt-conditioned and drive vision-language alignment. This split suggests that treating sinks as a uniform effect is incomplete. Building on this insight, we propose SAGE (Sink-Aware Guided Emphasis), a lightweight intervention that steers decoder attention away from PIS and toward query-dependent regions of interest (ROIs) using token-aligned ROI masks derived from standard vision backbones such as CLIP, ViT, and DINOv3. Evaluated on diverse vision-encoder + decoder-only LLM VLM families, SAGE improves visual grounding, reduces hallucinations, and yields consistent gains across public downstream vision-language benchmarks, including fine-grained visual discrimination settings where localized evidence is crucial, when instantiated with backbone-derived ROI masks.