Pairwise preference cycles improve video generation reinforcement learning
When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO
Computer Vision and Pattern Recognition
Summary
When training AI to create videos, it can be tricky to judge which videos are better. The authors noticed that when comparing videos in pairs, preferences often cycle—meaning video A is preferred over B, B over C, but then C over A. This cycle can confuse the training process. To fix this, they developed a new method that looks at these cycles to better understand overall preferences without needing more comparisons. Their approach improved video generation quality in tests and human evaluations.
What this means in practice
- •For video generation engineers: Improve reinforcement learning for video models by handling cyclic pairwise preferences to get more reliable reward signals.
- •For machine learning practitioners: Use graph-aware advantage aggregation methods to enhance preference-based reinforcement learning in settings where feedback has cycles.
Authors
Xinrui Lin, Fan Li, Shumin Wang, Dehong Kong, Yanyong Zhang
Abstract
Pairwise preference feedback has become a widely adopted supervision signal for reinforcement learning in video generation models, as pairwise comparisons provide a more reliable alternative than pointwise scalar ratings for evaluating the quality of generated videos. Specifically, we define a preference graph for each GRPO group under a pairwise evaluator, where nodes represent sampled rollouts and edges encode pairwise preferences. Auditing 6,976 on-policy comparison graphs, we find that cyclic preferences are prevalent and persist throughout training. Naive pairwise reward aggregation based on average win rates discards the underlying comparison structures and treats each rollout independently, which may introduce unreliable advantage estimates for certain rollouts and consequently affect reinforcement learning optimization. Therefore, we propose the \emph{Reliability-Anchored Preference Walk} (\ours{}), a graph-aware advantage aggregation method that strengthens the global preference signal from win rates with reliable relational evidence from cyclic preference graphs through a reliability-guided preference walk. Under matched evaluator, comparison protocol, group size, prompt schedule, and query budget, \ours{} requires no additional evaluator queries beyond the observed comparisons within each rollout group and improves over Pairwise-GRPO across controlled comparisons, transfer settings, and blind human evaluations.