Vision language models improve perception by adapting visual processing depth
Loop to Look Better: Learning Adaptive Depth for Question-Guided Perception in Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Models that combine images and text often look at pictures just once before answering questions about them. This can limit how well they understand because they don't reconsider the image with the question in mind. The authors propose a method called LoopLook, which lets the model look at parts of the image multiple times, refining what it sees based on the question. This approach helps the model give better answers while only slightly increasing the time it takes to start its response.
What this means in practice
- •For computer vision engineers: Enhance VLMs to adaptively reprocess visual features for better question answering accuracy across varying task difficulty.
- •For mobile device developers: Improve visual question answering systems that maintain performance under low-resolution inputs with minimal latency increase.
Authors
Wencheng Ye, Anning Hu, Hongye Fang, Tianyi Wang, Junchi Yan
Abstract
Visual perception should be iterative and task-directed, with representations refined in response to the question and the need for further processing. While Vision-Language Models (VLMs) increasingly support extended reasoning, their visual tokens are still typically acquired in a single, task-agnostic encoder pass. This creates an evidence-acquisition bottleneck: models may reason longer without refining the visual evidence needed to answer the question. We propose LoopLook, a lightweight encoder-side recurrent perception framework that equips pretrained VLMs with adaptive visual depth. LoopLook recurrently revisits a middle-to-late window of the vision encoder, injecting question representations at each loop to guide visual refinement. A learned stopping mechanism adaptively determines the recurrent depth, while a token-wise Mixture-of-Looks (MoL) combines representations from different loops, allowing each visual token to selectively integrate information acquired across recurrent steps. Across four backbones and eight benchmarks, LoopLook consistently improves performance over the baseline, yielding gains of up to 8.6% on MMStar and 8.1% on HallusionBench. Our analysis shows that LoopLook progressively refines visual attention and adaptively allocates depth according to task difficulty. It also better preserves task-relevant visual information under reduced input resolution, while increasing time to first token (TTFT) by only 5.1-6.5%. By bringing recurrence from reasoning back into perception, LoopLook transforms visual perception from a one-shot glimpse into an adaptive process of recurrent refinement.