Reward method improves fact-based response precision in ai models
Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL
Computation and Language
Summary
Reinforcement learning models often try to cover all parts of a given rubric but end up giving longer and sometimes incorrect answers. The authors found that focusing only on coverage (recall) leads to overly detailed and less accurate responses. They created a new reward called FORK that balances coverage with precision by ensuring each part of the answer is necessary and factually correct. Their tests show this approach makes responses shorter, more precise, and preferred by users over base models.
What this means in practice
- •For ai developers: Improve AI-generated responses in tasks requiring accuracy and conciseness using a balanced reward that mitigates verbosity and errors.
- •For medical ai teams: Enhance clinical AI assistants’ response quality by applying a reward method that increases factual correctness while maintaining coverage.
Authors
Hyun Ryu, Jong-Heon Kim, Yiqing Xie, Seungone Kim, Carolyn Rose, Eunho Yang
Abstract
Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench).