Vision encoder adapts computation for better visual question answering
AdaLoop: Thinking with the Vision Encoder
Computer Vision and Pattern Recognition
Summary
Most vision-language models look at an image once to answer a question, but some questions need more detailed or repeated image analysis. The authors introduce AdaLoop, a method that lets the vision encoder think multiple times about the image depending on the question, improving reasoning. They also created VisRO, a set of tests to measure how well models handle visual reasoning. AdaLoop worked better than other methods on various tests, even when questions or images were different from what the model had seen before. This suggests that vision components can do more complex thinking when guided by the question.
What this means in practice
- •For computer vision engineers: Implement question-adaptive visual processing to improve image understanding in interactive AI systems.
- •For machine learning platform developers: Incorporate iterative vision encoder methods to enhance generalization of vision-language models across diverse tasks.
Authors
Xinran Nicole Han, Yiyang Lu, Katelyn Gan, Todd Zickler
Abstract
Most vision-language models (VLMs) treat the vision encoder as a static feature extractor: it encodes an image once, independently of the question, and leaves subsequent reasoning to the language model. Yet different questions may require different visual operations and processing depths. We present AdaLoop, a framework for question-conditioned visual recurrence in VLMs. AdaLoop adapts visual computation to the question and reuses the encoder's transformations across multiple iterations, with the pretrained vision weights frozen. We also introduce VisRO, a procedural dataset and generation pipeline for training and evaluating vision-centric reasoning. On six VisRO tasks, AdaLoop outperforms standard fine-tuning and question-conditioned single-pass baselines with matched training data and optimizer updates. These gains hold across three VLM families on both in-distribution and out-of-distribution examples. Our findings suggest that pretrained vision encoders can contribute to reasoning through task-directed, iterative visual computation.