Multimodal models improve reasoning using detailed token-level rewards
Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
Artificial Intelligence
Summary
Multimodal models that combine text and images struggle with reasoning because they get feedback only for entire answers, not for the detailed steps they use. The authors found that correct reasoning shows certain patterns when looking at each word token's connection to the image and word uncertainty. They created a new method to give detailed feedback at each token based on these patterns, helping models learn better. This method improves performance and learning stability on multiple tests involving reasoning over images and text.
What this means in practice
- •For multimodal ai developers: Improve training of AI models that answer questions using both images and text by providing detailed feedback on each reasoning step.
- •For interactive ai interface builders: Build AI systems that produce more accurate and explainable answers by reinforcing token-level understanding of visual information.
Authors
Zhihan Zhang, Lizi Liao
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet existing frameworks rely on coarse, sequence-level reward signals that lack the fine-grained supervision over the visually-grounded steps within a multimodal reasoning chain. We investigate this gap through the lens of two token-level metrics: visual dependency (i.e. how much a token's prediction relies on the input image features) and predictive entropy. Our empirical analysis reveals two key findings: (1) correct reasoning chains exhibit a markedly sharper entropy reduction as visual grounding intensifies, compared to incorrect ones; (2) pivotal tokens, those whose misprediction triggers reasoning collapse, are statistical outliers in the joint distribution of visual dependency and predictive entropy derived from correct chains. Motivated by these findings, we propose token-level perception-grounded advantage estimation (TPAE), which estimates token-level advantages by measuring each token's statistical consistency with the vision-entropy patterns of correct rollouts. TPAE leverages this granular score to modulate the sequence-level advantage, producing a fine-grained supervision signal that can be integrated into various RLVR frameworks. Extensive experiments on seven benchmarks show that TPAE consistently outperforms leading strong baselines, yielding more stable and efficient optimization for multimodal reasoning. The code is publicly available at https://github.com/Zhihan72/TPAE.