Video reasoning improved by adaptive token selection and counterfactual learning
Beyond Entropy: Self-Diagnostic Multi-Role Token Optimization for Video Reasoning
Computer Vision and Pattern Recognition
Summary
Figuring out which parts of a video help answer questions is hard because some details might seem important but aren’t. The researchers created a new way called DyCPO that helps computers focus on the right visual clues by learning from their own successes and mistakes. This method balances exploring new ideas with zeroing in on the parts of the video that really matter for giving correct answers. Their tests show this approach helps computers understand videos better and make smarter decisions when reasoning about them.
What this means in practice
- •For video analytics teams: Enhance automated video analysis systems by improving token-level focus on relevant visual features for better reasoning outcomes.
- •For multimedia content developers: Create smarter interactive video applications that adaptively identify important content while minimizing irrelevant visual noise to improve user experience.
Authors
Yudong Han, Yong Wang, Zaiquan Yang, Liang Lin, Chongyang Tao, Xiangxiang Chu, Liyuan Pan
Abstract
Reinforcement learning with verifiable rewards has substantially advanced multimodal reasoning, yet it remains fundamentally limited by ambiguous token-level credit assignment. While high-entropy token heuristics encourage possibility exploration, naively extending them to video reasoning tends to induce lengthy reasoning, as the model becomes overly reliant on high-entropy visual activations. Alternative approaches that rely on counterfactual-based visual token localization for credit assignment also tend to over-prioritize visual exploration at the expense of decisive reasoning cues for answer derivation, thereby exacerbating the interference from spurious visual nuances. Moreover, these methods employ static counterfactual strategies that fail to co-evolve with the policy during training. In this paper, we introduce DyCPO, a co-evolutionary framework that jointly optimizes reliable token selection and adaptive counterfactual intervention. It constructs a multi-role dependence metric to balance visual exploration and answer-relevance mining in token-wise contrastive learning, while suppressing exploration-only filler tokens and spurious visual noise. Rather than relying on static counterfactual priors, DyCPO dynamically derives counterfactual signals from the model's own successful and failed rollouts, enabling self-diagnostic analysis and co-evolution of the optimization objective with the policy. Extensive experiments on complex video reasoning and general video understanding benchmarks demonstrate consistent performance improvements, establishing DyCPO as a robust token-level credit assignment paradigm for multimodal reinforcement learning.