Vision models adaptively pick layers for better visual reasoning
Mixture of Layers: Dynamic Layer Routing for Visual Reasoning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Many AI systems for understanding images rely on only the last parts of their visual processing, missing details like small objects or text. The authors created a method called Mixture of Layers (MoL) that lets these systems pick which parts of the visual processing to use depending on the question asked. This approach helps the AI focus on the right details to answer visual questions more accurately. Tests showed MoL improved performance a lot on tasks needing fine visual details without extra inputs or larger data. The work suggests smartly choosing visual information inside AI systems is key to better seeing and reasoning.
What this means in practice
- •For visual ai developers: Improve AI image understanding by dynamically selecting internal visual features based on the specific visual query, boosting accuracy on detailed tasks.
- •For augmented reality engineers: Enable AR systems to better recognize subtle and small visual elements by routing through relevant processing layers dynamically.
Authors
Jeonghwan Kim, Sofia Stoica, Jiwan Chung, Ansel Blume, Hyeonjeong Ha, Zhenhailong Wang, Xin Luna Dong, Heng Ji
Abstract
Pre-trained vision encoders contain layer-wise visual representations that differ in spatial granularity, semantic abstraction, and sensitivity to local details. However, most Multimodal Large Language Models (MLLMs) rely on only the final or penultimate vision encoder representations or fixed aggregation rules, making visual abstraction largely query-agnostic and limiting access to fine-grained cues such as small objects, spatial details, text, and subtle visual attributes. In this work, we propose Mixture of Layers (MoL), an instruction-conditioned layer routing approach at the visual patch level that dynamically aggregates query-relevant latent representations from intermediate vision encoder layers. Given a text query, MoL predicts routing probabilities over vision encoder layers and performs a top-k sparse aggregation over selected hidden states at either the image level, patch level, or through a hybrid routing mechanism. In doing so, MoL enables query-adaptive access to layer-specific visual features for fine-grained visual reasoning. Our experiments across 7 fine-grained visual reasoning tasks demonstrate substantial performance improvements, especially across fine-grained visual grounding and understanding tasks such as +18.9% improvement on V* in overall accuracy, +4.5% on HRBench4K, and +16.3% on CharXiv compared to the baseline MLLMs, without resorting to multi-resolution inputs, simple interleaving of multiple vision encoders, or increasing the number of patch tokens. We study vision encoders' receptive field scales across different layers and their sampling behaviors to provide an in-depth analysis of why layer-wise sampling is helpful, demonstrating that conditional visual representations are a key step towards better visual perception and reasoning in MLLMs. Our project page is available at https://wjdghks950.github.io/mol.github.io/.