RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning
2026-08-10 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors address the difficulty of training large language models to meet multiple criteria in open-ended tasks by proposing RISE-RL, a method that focuses on learning from examples where certain rubric criteria are often missed. Instead of treating all feedback as a single number, their method filters and re-tests promising behavior to better target areas where the model is weak. They show that RISE-RL improves model performance across various tasks like writing, chat, health, and science, outperforming previous approaches especially with larger models. Their approach also helps models produce more diverse and higher-quality creative writing.
Large Language ModelsReinforcement LearningRubric-based FeedbackOpen-ended TasksPolicy OptimizationSelective ExplorationReward FilteringAuxiliary ObjectivesCreative WritingModel Alignment
Authors
Jinkun Hou, Zhuo Liu, Huimin Ren, Hongsheng Xin, Pan Zhou, Kun Zhan
Abstract
Aligning Large Language Models (LLMs) for open-ended tasks is challenging because responses must satisfy multidimensional criteria without following a single correct generation trajectory. Existing rubric-based reinforcement learning (RL) methods compress fine-grained criterion-level feedback into scalar rewards, making persistent capability gaps difficult to target under limited on-policy exploration. We propose $\textbf{RISE-RL}$ (Rubric-Informed Selective Exploration), which uses repeatedly missed rubric criteria to elicit privileged trajectories that are difficult to discover through unguided exploration alone. RISE-RL retains only trajectories whose complete-rubric reward exceeds the mean reward of natural rollouts, and then re-evaluates them under the original prompt to emphasize behaviors that remain weakly supported by the natural policy. The resulting guidance signal is optimized through a separate auxiliary objective and removed once its additional benefit diminishes. Experiments with 4B and 14B models across writing, chat, health, and science show that RISE-RL achieves the highest mean score on every evaluated benchmark under guidance-free evaluation. Compared with standard Rubric-RL, it improves the average score by 1.3 points at the 4B scale and $\textbf{3.3 points at the 14B scale}$, including a $\textbf{6.0-point}$ gain on CreativeWriting-V3. It also improves creative-writing diversity and yields gains on objectively scored medical and scientific benchmarks. These results indicate that selective internalization through reward filtering and policy support shaping is effective for open-ended reinforcement learning.