AI summaryⓘ
The authors studied vision-language models, which understand pictures and text together, but found that breaking images into fixed parts can hurt recognizing whole objects. They discovered that measuring uncertainty (entropy) in the model's attention helps identify important image regions connected to the question asked. Using this insight, they created ENCORE, which picks image crops with low entropy to keep relevant parts intact and trains the model to focus attention better on these parts. Testing on multiple benchmarks showed small but consistent accuracy improvements with minimal extra training. Their method improves how these models focus on important image details when answering questions.
Vision-Language ModelsTransformerEntropyCross-modal AttentionImage CroppingEntropy RegularizationVisual Question AnsweringPromptInferenceFine-tuning
Authors
Yuanhao Sun, Huawei Ji, Jiaxin Ding, Luoyi Fu, Xinbing Wang
Abstract
Vision-Language Models (VLMs) perform well on diverse vision-language tasks, but transformer-based visual encoders split images into fixed-resolution sub-images, compromising object integrity in lightweight VLMs. Existing methods only focus on the visual modality and fail to dynamically preserve the integrity of prompt-relevant regions, limiting performance. In this work, we observe that the early-layer image-text entropy of cross-modal attention strongly correlates with answer grounding quality and task accuracy. Building on this finding, we propose \textbf{ENCORE}, an entropy-guided framework with two components: At inference, an \textbf{Entropy-based Cropping Strategy} (ECS) evaluates a small set of candidate crops and selects the one with minimal entropy, preserving contiguous regions relevant to the prompt. At training, \textbf{Entropy Regularization Training} (ERT) augments next-token prediction with an entropy term that sharpens attention on key visual tokens while down-weighting irrelevant ones. Experiments on ten VQA benchmarks show that ENCORE, fine-tuning only 0.14\% of parameters, achieves an average 1.43\% accuracy gain and state-of-the-art performance among recent 2B-parameter VLMs. Our code is released in https://github.com/baokou-fw2/ENCORE.