Multimodal language models adaptively highlight key visual and textual evidence
Layers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models
Artificial Intelligence
Summary
Large language models that handle images and text sometimes miss important clues needed to answer questions. The authors of this paper created a method called AREA that helps these models decide when and how much visual or textual information to focus on while answering. AREA adjusts its highlighting dynamically during the answer generation, improving the model's accuracy without extra training. This method was tested on multiple benchmarks and showed better performance than other ways that highlight evidence without retraining.
What this means in practice
- •For ai engineers: Improve multimodal AI systems’ accuracy by dynamically selecting relevant image and text evidence during question answering.
- •For software developers: Incorporate adaptive evidence highlighting to enhance the performance of apps using pretrained language models with images and text.
Authors
Zhenbin Wang, Lei Zhang, Lituan Wang, Wei Huang, Yan Wang, Zhenwei Zhang
Abstract
Multimodal large language models (MLLMs) can answer knowledge-intensive visual questions by combining visual evidence from images with facts retrieved from external sources. However, MLLMs may overlook relevant evidence in both modalities, attending weakly to the textual sentences or visual regions needed for the correct answer. Recent efforts address this by highlighting retrieved text and marking visual regions before generation, but apply a fixed, one-shot policy that cannot adapt to three sources of variation: whether highlighting is necessary, how much evidence different examples require, and when different textual evidence becomes relevant as the answer unfolds. We introduce Adaptive Relevance-guided Evidence Allocation (AREA), a training-free inference-time method that formulates evidence highlighting as adaptive allocation. AREA generates a single probe token to read visual and textual relevance from fixed backbone layers, then makes three decisions: i) whether to intervene (controlled by natural attention coverage and visual sink contamination), ii) how much evidence to expose (determined by relevance entropy), and iii) when to refresh text during generation (triggered by causal context-attention peaks). Across four KB-VQA and seven standard multimodal benchmarks with nine frozen MLLM checkpoints, establishes the best performance among training-free highlighting methods.