HarnessRL improves open-ended reinforcement learning with adaptive training
HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL
Computation and Language
Summary
Training AI models to generate responses based on language criteria can get stuck when the scoring system stops working well. The authors of this paper found that fixed methods for picking tasks and judging responses can cause scores to be unhelpful or misleading as the model learns. They created HarnessRL, a system that evolves alongside the model by adjusting task selection, exploration, and evaluation based on problems it sees during training. This approach helped models learn more effectively and faster across various tasks like medical advice, writing, and role-play.
What this means in practice
- •For ai model trainers: Improve training of language models for complex, open-ended tasks by dynamically adjusting task difficulty and reward evaluation during learning.
- •For customer support developers: Build more reliable AI agents for diverse support tasks by enabling adaptive feedback and task selection during training.
Authors
Xiaoqiang Wang, Ting-En Lin, Yuchuan Wu, Pinyi Zhang, Caixin Kang, Ling Li, Bingxue Qiu, Jieping Ye, Yongbin Li, Bang Liu
Abstract
Rubric-based reinforcement learning (RL) trains policies for open-ended generation with natural-language criteria as rewards, but these rewards can become ineffective as training progresses. We identify two failure modes. Under non-discriminative rewards, scores collapse to uniformly low or high values due to insufficient exploration or a mismatch between task difficulty and policy capability. Under spurious rewards, higher scores do not correspond to better responses. Both reflect a training configuration whose task sampling, exploration, and evaluation stay fixed as the policy evolves. We introduce HarnessRL, a training-time harness that co-evolves with the policy through three interfaces for task sampling, rollout guidance, and rubric criteria. HarnessRL maintains a training-time memory from which a harness evolver retrieves within-task history and cross-task evidence. Using this evidence, the evolver attributes each reward failure to the interface whose update can address it. It then runs a proposer and just-in-time critic loop in which the critic accepts a candidate update only after measuring its effect on the current frozen policy, returning rejected candidates for revision. We also curate Rubric-Mix-10k, a five-domain training and evaluation suite for open-ended RL across medical, science, writing, role-play, and instruction-following tasks. On Rubric-Mix-10k, HarnessRL outperforms the strongest baselines by 2.8 to 3.1 points across both 4B and 9B model scales and nearly halves the training steps to its best checkpoint. Code and data are available at https://github.com/Tongyi-ConvAI/Qwen-Character/tree/main/HarnessRL.