MetaRubric improves reinforcement learning accuracy for medical QA tasks
MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning
Artificial Intelligence
Summary
Rubric-based reinforcement learning helps AI systems get partial credit for complex tasks by judging how well they meet detailed criteria. But sometimes, these systems give credit even when the required information is missing, which can mislead the learning process. The authors introduce MetaRubric, a new method that adjusts how rewards are given by carefully checking if the response truly contains evidence for each criterion and then fine-tuning the reward rules based on the AI’s own answers. This approach leads to better accuracy in medical question-answering tasks compared to previous methods.
What this means in practice
- •For medical ai developers: Improve the accuracy of AI systems answering medical questions by refining reward signals to better match evidence in responses.
- •For multimodal ai engineers: Enhance training of AI models that integrate multiple data types in healthcare by adaptively adjusting rubric-based rewards.
Authors
Yuxuan Fan, Jaehong Yoon
Abstract
Rubric-based reinforcement learning extends reward-driven optimization to open-ended tasks by assigning partial credit to individual response requirements. However, rubric judges can assign a high criterion score even when the information or action it requires is absent from the response, a failure mode we term Vacuous Credit. Such awards persist after the required information is removed and can reverse the sign of a response's GRPO advantage. To address this problem, we introduce MetaRubric, which alternates evidence-aware policy optimization with response-guided rubric adaptation. We construct counterfactual counterparts by changing one task-relevant fact in each prompt. During policy optimization, credit is assigned only when the response contains sufficient evidence to satisfy the required rubric criterion. After each policy-optimization stage, current policy responses guide revisions to original and counterfactual criteria while preserving the meaning of the original prompt's initial rubric as interpreted under each prompt's facts. We also adapt criterion weights at stage boundaries to better address observed policy errors. Across multiple backbones, MetaRubric improves PubMedQA accuracy by 6.00--20.40 percentage points over static-judge GRPO, with further gains on HealthBench-Hard and two multimodal medical benchmarks.