Vision language model learning improves by focusing on visual signals
Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning
Artificial Intelligence
Summary
Large vision-language models struggle to keep track of important visual clues when learning from their past experiences. The authors propose a new learning method called PIVOT that helps these models remember and reinforce the most important visual reasoning steps. PIVOT uses a special replay system to revisit useful past experiences and allocates more credit to tokens that have strong visual support and impact on reasoning. Experiments show that this approach helps the models reason better across multiple tasks involving vision and language.
What this means in practice
- •For ai developers: Improve vision-language AI systems by enhancing reasoning with better use of visual data during training.
- •For robotics engineers: Develop robots that understand and process visual and language input more reliably for tasks requiring step-by-step reasoning.
Authors
Xinxin Song, Siyuan Li, Tingxiong Xiao, Jinli Suo
Abstract
Reinforcement learning with verifiable rewards (RLVR) has significantly improved the reasoning capabilities of large vision-language models (LVLMs). However, standard on-policy RLVR algorithms face a critical optimization bottleneck in preserving and reinforcing visually grounded reasoning behaviors: valuable visually-grounded reasoning trajectories are discarded after a single update, while uniform token advantage allocation prevents the model from reinforcing critical perception or reasoning steps. To bridge this gap, we propose PIVOT, a dual-level learning framework that anchors policy optimization around informative visual reasoning signals. Specifically, PIVOT introduces a self-calibrated experience replay mechanism, which selectively collects and replays visually-grounded historical experiences as stable reference anchors for policy optimization. Building upon this, we further design a vision-guided advantage allocation mechanism to allocate additional vision-aware advantages to tokens based on their local visual support and impact on downstream reasoning. Extensive experiments across diverse benchmarks demonstrate that PIVOT achieves highly competitive performance in enhancing the multimodal reasoning capabilities of LVLMs.