Video captioning improves with detailed multi-step reward learning
MVRCap: Towards Comprehensive Video Captioning via Hierarchical Multi-Dimensional Verifiable Rewards
Computer Vision and Pattern Recognition
Summary
Creating accurate and detailed descriptions for videos is hard because computers usually learn from simple feedback that doesn’t fully capture everything in the video. The authors designed a new system called MVRCap that teaches computers to write better captions by giving them more detailed and careful feedback along several aspects, such as covering many things in the video, making sure the facts are right, and keeping captions clear and concise. They also created special datasets to help the system learn. Their experiments show this method works better than previous ones on popular video captioning tests.
What this means in practice
- •For video platform developers: Improve video description generation systems with structured multi-dimensional reward feedback to produce more comprehensive and accurate captions.$Commercial implications: Enables new or enhanced commercial captioning services that provide richer, more reliable video descriptions for viewers, improving accessibility and engagement.
- •For media content curators: Use improved video captions to better organize and search video archives with deeper semantic coverage and factual correctness.
Authors
Weihuang Lin, Xianing Chen, Zhibin Wang, Congzhi Zhang, Yiwei Ma, Jiayi Ji, Miao Chen, Jun Song, YuCheng, Bo Zheng, Xiaoshuai Sun, Rongrong Ji
Abstract
Despite significant progress, current models struggle to generate comprehensive video captions, constrained primarily by coarse-grained supervision, which often fails to capture spatial-temporal dynamics and the underlying logical intent. To address this issue, we introduce MVRCap, a novel framework that reformulates video captioning as a reinforcement learning task steered by structured verification. Central to our approach is a Fine-Grained Verifiable Reward mechanism, serving as dense supervision to enforce rigorous visual-semantic alignment. Specifically, we construct this reward system anchored by three complementary pillars: a Hierarchical Concept Coverage Reward utilizing our newly constructed MVRCap-QA dataset to ensure multi-dimensional completeness and semantic depth, a Factuality Verification Reward to enhance factual consistency, and a Conciseness Constraint to maximize semantic density. Furthermore, we curate the high-quality MVRCap-SFT dataset, providing structured annotations across six key dimensions, to serve as a robust foundational initialization for the RL policy in this training pipeline. Extensive experiments demonstrate that MVRCap achieves state-of-the-art performance on authoritative benchmarks including VDC, CAPability, and VC4VG.