Policy and rubric co-evolve to improve reward guidance in learning
Reward-Adaptive Co-Evolution of Policy and Rubric
Artificial Intelligence
Summary
Getting computers to learn good behaviors often relies on rewards that tell them how well they did. Simple one-number rewards can miss important details, so the paper’s authors designed a system that adjusts both the learning rules (the rubric) and the computer’s approach (the policy) together. Their method decides when to update each part based on how helpful the rewards are, leading to better and more balanced learning outcomes. They tested it on multiple tasks and found it outperformed other methods that use fixed schedules for updating.
What this means in practice
- •For reinforcement learning developers: Enhance the effectiveness of reward-based training by adapting both the scoring rubric and learning policy dynamically to maintain discriminative feedback.
- •For game ai designers: Improve in-game agent learning by using reward-adaptive rubric methods to better capture multidimensional player responses and evolving agent behavior.
Authors
Yizhu Chen, Jianguang Jiang, Jiameng Huang, Chuyu Fang, Bojun Zhang, Zhenyu Jiao, Xing Yuan, Tong Lin
Abstract
Rubric-based rewards provide fine-grained, interpretable supervision for open-ended reinforcement learning, where holistic scalar rewards struggle to capture multidimensional response quality. Static rubrics can lose discriminative power as the policy distribution evolves, while existing policy--rubric co-evolution methods rely on globally prescribed update schedules that ignore heterogeneous reward informativeness. We introduce RACE, a Reward-Adaptive Co-Evolution framework. At the group level, per-rollout-group training-signal routing classifies groups as informative, diversity-collapsed, or rubric-failed, updating the policy only with informative groups while caching rubric failures. At the phase level, event-triggered phase switching enters rubric optimization upon persistent reward saturation and returns when rubric-reward improvement plateaus. Rubric phases train the generator on cached pairs using a frozen judge and a scale-normalized pairwise-margin reward. Across both levels, online threshold adaptation adjusts routing and switching thresholds through proportional control. Under a unified evaluation protocol, RACE achieves the highest average performance and outperforms the strongest fixed-alternation baseline on all eight tasks, while improving rubric discrimination.