RewardVerse stabilizes video scoring for better reinforcement learning
RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Scoring videos for quality during training can be tricky because simple scores often change meaning depending on the video, making learning unreliable. The authors propose RewardVerse, which breaks down video evaluation into clear criteria before scoring, so the scores stay consistent and meaningful. They also develop a two-step training method that teaches the system to create useful criteria and reliable scores based on human ratings. This approach improves the stability and accuracy of video quality scores, helping train better video-generating AI.
What this means in practice
- •For video generation engineers: Improve training consistency of video generation AI by using rubric-based reward models that reduce score instability.
- •For machine learning platform builders: Integrate rubric-guided reward functions into reinforcement learning workflows for more interpretable and stable optimization signals.
Authors
Zhenchen Tang, Yang Li, Songlin Yang, Bo Peng, Xiaotong Zhao, Shuai Li, Haotian Fan, Alan Zhao, Jing Dong
Abstract
Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.