Efficient visual reasoning by adapting memory to changing evidence
Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning
Computer Vision and Pattern Recognition
Summary
High-resolution images create huge amounts of visual information that can be hard for computers to process during multi-step reasoning tasks. The authors developed ViMoD, a new method that keeps a small summary of an image but can also pull out detailed parts of the image when needed as reasoning changes. This method uses special grouping and selection processes to remember and retrieve important image details without looking at everything all the time. ViMoD was tested on several reasoning tasks and performed better than previous methods while adding almost no extra model size.
What this means in practice
- •For computer vision engineers: Enable visual reasoning systems to dynamically access detailed image regions during complex multi-step tasks while using fewer computational resources.
- •For mobile device developers: Create apps that perform advanced image understanding with less memory and processing power by selectively focusing on important image details.
Authors
Yicheng Xue, Han Wu, Jufeng Yang, Minjing Dong, Xinghao Chen, Hanting Chen, Jianyuan Guo
Abstract
Processing long visual token sequences from high-resolution images makes multi-step reasoning computationally expensive for multimodal Large Language Models (MLLMs). Existing one-shot pruning and aggregation methods compress visual tokens into a fixed context before decoding. However, visual evidence needs can shift as reasoning unfolds, making it difficult for a fixed compressed context to retain all the details needed across stages. To address this challenge, we propose ViMoD, a lightweight framework that maintains a compact visual context while preserving access to original fine-grained evidence as reasoning needs evolve. Deformable Aggregation of Region-wise Tokens (DART) learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens. Temporal Routing for Adaptive Contextual Evidence (TRACE) integrates decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. Selected Fine tokens augment the persistent Coarse context in the frozen backbone, enabling stage-specific evidence access without continuously attending to all visual tokens. On Qwen3-VL-4B, ViMoD outperforms all evaluated baselines on all eight reasoning benchmarks at a 20% target visual token budget, improving the mean normalized score by 39.0% over the strongest evaluated one-shot baseline. These gains are achieved with only 0.0546% additional trainable parameters relative to the frozen backbone.