Reinforcement learning reward systems evolve alongside policies for better results
EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning
Artificial Intelligence
Summary
Sometimes in training AI agents, the rules for rewards stop working well because the agents find ways to trick the system or the rewards become unclear. This paper shows a way to let the reward system change and improve as the agent learns, making the training more reliable. The authors created EvoRS, which updates rewards based on the agent’s own experiences and helps reduce problems like cheating the reward or missing important feedback. Their tests show that evolving rewards give better performance than fixed reward systems in open-ended tasks like writing and roleplaying.
What this means in practice
- •For game developers: Improve non-player character behavior by evolving reward systems during training to reduce unintended exploits and increase engagement.
- •For dialog system engineers: Train chatbots with evolving reward criteria to maintain response quality and decrease manipulation of reward signals over long training periods.
Authors
Weiyuan Li, Aili Chen, Xintao Wang, Yikai Zhang, Qingqing Dong, Jinghan Xu, Hongru Hou, Wenxuan Zhao, Chengkun Lang, Jun Gao, Yuanli Guo, Hongcheng Guo, Yanghua Xiao, Deqing Yang
Abstract
Open-ended reinforcement learning often relies on rubric-based rewards for tasks without directly verifiable answers. Yet the policy and reward system form a dynamic feedback loop: as the policy optimizes the current reward, an initially useful reward system may become unreliable due to reward hacking or reduced response discriminability. The reward system should therefore evolve rather than remain fixed during training. Existing dynamic-rubric methods adapt evaluation criteria, but reward failures can also arise from scoring mechanisms or signal composition. We introduce EvoRS, a self-evolving RL framework that evolves the reward system from on-policy experience, representing it as an executable Reward-DAG. Specifically, an agentic designer updates this system from on-policy rollouts and reward traces to maintain train-time reliability. Across writing and roleplay, EvoRS achieves the best quality under all three judges, outperforming the policy by \(2.107\) and \(4.767\) points, respectively, while reducing reward hacking and coverage failures and preserving reward informativeness. Ablations confirm that a comprehensive fixed reward system cannot remain reliable in open-ended tasks and must evolve throughout training.