Reinforcement learning improves visual reasoning by verifying token corrections
Reinforcement Learning with Visually Attributable On-Policy Self-Distillation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
It is hard for AI systems to judge how well they understand visual information when they rely only on broad feedback about whether they completed a task correctly. The authors show that adding detailed advice on individual parts of the visual input can help, but only if this advice truly reflects the visual evidence instead of side influences like language hints or style. They propose a new method called Visual-Attribution Gating that checks token-level corrections to make sure they are supported by visual information before using them to improve learning. Their experiments show this approach leads to better performance across many visual reasoning tests.
What this means in practice
- •For autonomous vehicle engineers: Enhance visual decision-making by incorporating visually validated token corrections into reinforcement learning models for more accurate environment interpretation.
- •For robotics software developers: Improve robot perception by using evidence-grounded token-level feedback during training for better task success in visually complex scenarios.
Authors
Shiao Xie, Siyu Chen, Jianwei Lv, Xiandong Li, Yujin Wang
Abstract
Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous token-level updates. To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence. We instantiate this paradigm with Visual-Attribution Gating (VAG), which uses the verifier-derived advantage as the optimization anchor and validates each privileged correction against the contrast between evidence-present and evidence-degraded teacher views. By admitting only corrections directionally supported by this visual contrast, VAG preserves visually grounded guidance while suppressing inconsistent residuals. Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.