Reinforcement learning improves open-ended content with evolving rubrics
RARE: Reinforcement Learning with Adaptive Rubric Evolution for Open-Ended Generation
Computation and LanguageMachine Learning
Summary
Some tasks, like writing stories or videos, are hard to judge automatically because there's no single right answer and many things to consider. The authors introduce a way for AI to learn better by changing how it scores these tasks as it gets better, focusing on what still needs improvement. This method, called RARE, adjusts the scoring criteria dynamically, helping the AI improve more than fixed scoring rules. They tested it on generating video scripts and other tasks, showing better performance and preferences from humans.
What this means in practice
- •For content production teams: Generate coherent and constraint-following video scripts with improved quality scoring that adapts during training to fix reported flaws.
- •For natural language generation developers: Train AI models for writing and generation tasks using adaptive rubrics that improve how multiple quality aspects are balanced and scored.
Authors
Xueyun Tian, Bingbing Xu, Chuyue Li, Jiaming Liu, Chao Zhang, Weijia Feng, MingHua Ma, Yu Zhao, Jingchu Wang, Ruihua Huang, Yuanzhuo Wang, Huawei Shen
Abstract
Verifiable rewards have advanced reinforcement learning in mathematics and code. However, many valuable open-ended tasks share three properties: no automatic verifier of overall quality, multiple requirements to satisfy jointly, and multiple valid responses to each query. Training therefore needs feedback on requirement satisfaction across different valid responses. Rubric-based reinforcement learning provides this feedback by aggregating scores from explicit criteria. As learning progresses unevenly, some criteria become fully satisfied and lose discrimination while others remain informative. Dynamic methods revise shared rubrics or select evolving criteria to restore discrimination, but may also change useful criteria. To address this, we propose Reinforcement Learning with Adaptive Rubric Evolution (RARE). The core idea is to use current policy feedback to guide both which criteria change and what their replacements assess. Sample-level updates replace criteria fully satisfied across current outputs with checks of remaining quality differences, retaining all other criteria and scores. Within this rule, we compare rewarding desirable qualities with penalizing observed flaws. Video script generation exemplifies these properties and supports content production. Its demands for coherent, executable narratives under constraints make it a challenging testbed. Targeting flaws performs best across three video types with a mean gain of 3.82 points over matched static rubrics. Model and human preferences support these gains. Improvements also extend to writing and general generation benchmarks.