AI summaryⓘ
The authors introduce a method called Multi-scale Adaptive Vision Encoder (MAVE) to improve how vision-language models understand images. Instead of using only the final visual features, MAVE combines information from different layers of a vision transformer to keep details like text and edges. It smartly selects important image tokens based on the question asked and image complexity, reducing the amount of data processed and speeding up response time. The authors also propose techniques to keep the quality of the compressed image information. Their simulation shows MAVE uses fewer tokens, improves accuracy on visual question answering tasks, and runs faster, though these results are preliminary and need real training and testing.
Vision-language modelsVision transformerToken routingFeature fusionVisual token compressionTextVQAVQAv2Representation distillationSpatial diversity regularizationInference latency
Abstract
Vision-language models commonly project all tokens produced by a pretrained vision encoder into a large language model. However, final-layer features can discard text, local attributes, and spatial relationships, while high-resolution inputs substantially increase context length and inference latency. We introduce \method, a Multi-scale Adaptive Vision Encoder. \method uses position-dependent gates to fuse shallow, intermediate, and deep features from a vision Transformer, preserving global semantics while enhancing edges, text, and local structure. It then performs question-conditioned token routing according to question relevance, local information content, global semantics, and spatial coverage, with a token budget that adapts to image complexity. To mitigate compression loss, we further introduce full-to-compressed representation distillation and a spatial diversity regularizer. In an illustrative simulation under a unified 7B language-model framework, \method reduces the average number of SigLIP-SO400M visual tokens from 729 to 146 (approximately 80.0\%) and improves the mean score on VQAv2, GQA, TextVQA, ScienceQA-IMG, and MMBench by 2.2 percentage points, while reducing single-image time to first token from 228\,ms to 129\,ms. We provide the full model design and evaluation protocol. All reported numbers currently serve only as placeholders for paper organization and experimental design; formal claims require real training runs, independent replications, and official benchmark evaluation.