Video reward optimization improves dynamics using out-of-distribution exemplars
GraftOOD: Grafting Out-of-Distribution Exemplars into Advantage-Weighted Rectification for Video Reward Optimization
Computer Vision and Pattern Recognition
Summary
Optimizing computer-generated videos to get higher scored rewards can sometimes make their movements unrealistic or less dynamic, a problem called motion collapse. The authors propose a method named GraftOOD that mixes in external example videos from different distributions during training. This approach helps the system learn better video dynamics without needing extra complex adjustments. The authors show that their method improves several measures of motion quality while maintaining reward performance.
What this means in practice
- •For video synthesis developers: Enhance video generation models to preserve natural motion dynamics by integrating supervised out-of-distribution video exemplars during reward optimization.
- •For machine learning engineers: Improve reinforcement learning environments with videos by reducing motion collapse using exemplar-based reward adjustment methods.
Authors
Zheming Fu, Ruizhe He, Wenbin Ning, Xiaoxiao Ma, Xuan Yang, Lianrui Mu, Haoyang Huang, Chang Liu, Lei Wang, Siming Fu
Abstract
Video reward optimization can increase proxy rewards while reducing video dynamics, a mismatch we identify as \emph{motion collapse}, including under a video-level motion-quality reward. We propose \textbf{GraftOOD}, which replaces part of the online rollouts in selected prompt groups with out-of-distribution video exemplars. A shared velocity-regression objective combines group-relative reward rectification with sequence-level exemplar supervision, requiring neither exemplar generation trajectories nor an additional supervised-loss coefficient. We characterize its pointwise optimal field as a source-posterior-weighted combination of the two source-specific target fields. We construct \textbf{GraftSet-Video}, containing 9,588 format-matched exemplars for 2,397 prompts, to supply external supervision. On Wan2.1-1.3B, GraftOOD raises Dynamic Degree from 4.86 to 44.41 relative to reward-only FlowAWR, while HPSv3 decreases from 8.10 to 7.32 and VisionReward remains close. At the main setting, GraftOOD leads nine of ten metrics among three exemplar-augmented methods. Ablations of exemplar frequency and within-group count reveal dynamics--reward trade-offs rather than uniform gains from increasing exemplar usage. These results support external exemplar supervision as a complement to scalar reward feedback, without equating higher measured dynamics with coherent, visually plausible motion in the generated videos. Code is available at \href{https://github.com/Fuzm7/FlowAWR}{GraftOOD}.