Rubric structure improves reinforcement learning for question answering
Rubrics Beyond Evaluation: Exploiting Rubric Structure in Reinforcement Learning
Machine LearningArtificial Intelligence
Summary
When computers learn to answer open-ended questions, rubrics are used to score their answers based on several clear criteria. The authors show that instead of just using the rubric scores as simple rewards, you can improve learning by understanding and using the structure behind those criteria. Their method, called RSRL, breaks down the skills behind the rubric, helps develop weaker skills, and tracks progress for each skill separately. This approach led to better overall answers and improvements in weaker areas when tested on scientific question answering tasks.
What this means in practice
- •For ai system developers: Improve training of question answering models by incorporating rubric structures for better skill development and overall accuracy.
- •For educational software engineers: Build evaluation tools that not only score answers but guide AI learners by explicitly modeling and improving individual capabilities.
Authors
Yuxun Huang, Yunyao Zhang, ZeLiang Chen, Yi-Ping Phoebe Chen, Junqing Yu, Zikai Song
Abstract
Rubrics are widely used to evaluate open-ended question answering by breaking overall quality into explicit criteria. These explicit criteria make rubrics a rich source of learning guidance, yet their potential for guiding the training process has been largely overlooked. Existing methods use rubrics mainly as direct feedback signals, without explicitly modeling the capabilities behind their criteria. This leaves their capability structure largely unused in guiding the training process. We introduce Rubric-Structured Reinforcement Learning (RSRL), an RL framework that unlocks the full potential of rubrics by exploiting their structure throughout the training process, rather than treating them simply as direct reward signals. At its core, RSRL builds on three key designs: (1) Rubric-to-Capability Structuring first recovers the capability structure hidden in rubric criteria, (2) Self-Sufficient Capability Learning then uses this structure to bootstrap underdeveloped capabilities into effective learning, (3) Capability-Residual Credit Routing finally preserves capability-specific gains during optimization. Experiments show that RSRL consistently improves both overall performance and underdeveloped capabilities, boosting downstream scientific question-answering performance by up to 7.43 over the initial policy and 7.60 over non-rubric structured RL baselines, showing that reinforcement learning can substantially benefit from the capability structure encoded in rubrics.