Visual memory boosts large language model reasoning with images
ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
When large AI models look at pictures and think through problems, they often forget important details from the images over time. This paper finds a way to help the AI remember key visual information by using two kinds of memory: one that holds general context about the whole scene and another that focuses on small, important parts. These memories help the AI bring back relevant visual clues exactly when it needs them, making its answers better and more precise. The authors tested their method on many image-based tasks and showed it works better than previous approaches.
What this means in practice
- •For ai software developers: Build AI systems that solve complex tasks involving multiple images more accurately by using targeted visual memory during reasoning.
- •For computer vision engineers: Optimize computational cost by reducing the number of visual tokens processed during image understanding while maintaining accuracy.
Authors
Hao Jiang, Zhanyu Guo, Chenwei Wu, Yichen Guo, Qizhe Zhang, Junchi Yao, Jixian Wu, Jinhao You, Kai Tang, Jiajun Cao, Tinghao Wang, Mengyu Wang, Leo Anthony Celi, Shanghang Zhang
Abstract
As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memory reintroduces visual evidence during reasoning. We conduct a controlled analysis of visual memory along three axes: curation, organization, and access. We find that local evidence benefits from global context, compact latent representations balance accuracy and visual-context cost, and the utility of memory access depends on the reasoning state. Guided by these findings, we propose ReMAP (Reasoning-Time Memory-Augmented Perception), which couples two complementary latent memories: a static, question-conditioned Global memory that preserves scene and cross-image context, and a dynamic Local memory that uses this context as an anchor while selecting and re-encoding region-level evidence according to the current reasoning state. Both memories return compact latent tokens inserted into the reasoning sequence, and a reinforcement-learning access policy trained with branched rollouts decides when to continue reasoning or invoke Global or Local memory. On ten benchmark families, ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, exceeding the strongest prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points. Across four backbone families, enabling memory access improves over the same trained model with memory disabled, and on shared V*Bench, CV-Bench-2D, and MuirBench questions ReMAP reduces the visual tokens entering the reasoning sequence by 51.0-76.8% relative to the native-resolution backbone. Further analyses show that Global and Local memory form distinct yet complementary latent representations. Together, these components restore the perceptual cycle by letting the reasoning state trigger targeted visual retrieval, with the retrieved evidence guiding subsequent reasoning.