Latent visual reasoning improves by aligning tokens with image evidence
Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidence
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
Summary
Latent visual reasoning helps AI systems think about images using hidden code instead of words. The authors found that these hidden codes often don’t focus on the important parts of an image that change the answer. They created a new method called ReaLVR to guide these hidden codes more closely using clues from the image itself. This made the AI better at reasoning and answering questions about images across different model sizes.
What this means in practice
- •For machine learning engineers: Implement visual-evidence supervision to improve image reasoning accuracy in large multimodal models.
- •For computer vision developers: Design visual reasoning components that better align internal model states with relevant image regions for improved interpretability.
Authors
Xi Xiao, Tianchen Zhao, Youngeun Kim, Zhuowei Li, Linghan Xu, Jiaye Wu, Zheng Zhang, Xiang Xu, Xuanbai Chen, Farhan Tejani, Jakub Zablocki, Julia Xu, Yifan Xing
Abstract
Latent visual reasoning (LVR) enables multimodal large language models (MLLMs) to perform intermediate computation in continuous latent tokens rather than expressing every reasoning step in words. However, unlike textual CoT, latent reasoning is not directly observable, making it difficult to supervise what latent tokens learn. In this work, we first conduct a thorough analysis of latent-token behavior and identify a latent evidence-credit gap: latent tokens respond only weakly to image perturbations that alter the correct answer. We hypothesize that this issue stems from the lack of explicit supervision during GRPO training. These findings suggest that a final-answer reward provides too little guidance on what visual evidence to preserve or how credit should be assigned across latent tokens. To bridge this gap, we propose ReaLVR, which brings visual-evidence supervision to the model's own free-running latent trajectories. ReaLVR contrasts correct and model-generated wrong answers to determine where stronger supervision is needed, and relevant and mismatched visual evidence to specify what to preserve. Across three model families, ReaLVR consistently outperforms evaluated LVR baselines, achieving the highest five-task average of 63.7% on Qwen2.5-VL-7B. Crucially, we are the first to scale visual reasoning in latent space, showing that our framework continues to deliver robust improvements at frontier model scales up to 235B. Further analyses show more question-sensitive latent-token positions, stronger alignment with relevant visual regions, and greater fixed-context dependence on the most attended latent tokens.