Reliability-first method reduces reward hacking in policy optimization
STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking
Artificial Intelligence
Summary
Sometimes when training AI systems, the measure used to judge progress can be tricked, leading to better scores without actually improving quality. The authors introduce a new method called STAR-GRPO that checks how reliable each measurement is and adjusts the training accordingly. This helps avoid fake improvements caused by unstable or misleading reward signals. They tested this method in language tasks and medical reasoning, showing it better aligns training with true quality and reduces misleading optimizations.
What this means in practice
- •For machine learning engineers: Improve training of AI models by reducing false progress from unstable or misleading reward feedback during policy optimization.
- •For health data scientists: Enhance medical AI systems by better aligning training rewards with true evaluation criteria, reducing overclaiming on proxy measures.
Authors
Wan Tian, Zhongyi Li, Xiang Xu, Minhao Zou, Yijie Peng, Fuzhen Zhuang
Abstract
Reward hacking occurs when policy optimization exploits a brittle reward interface or an overly permissive proxy objective, improving the training score without improving the underlying response quality. This phenomenon is amplified in group-relative policy optimization: an unsupported reward can shift the group baseline and alter the updates of other rollouts, while post-hoc or purely relative weighting cannot represent group-wide uncertainty. We propose \emph{Self-Tuned Anchored Reliability Group-Relative Policy Optimization} (STAR-GRPO), a reliability-first advantage estimator based on paired assessments of the same rollout. STAR separates the quality signal from its learning influence: score disagreement determines rollout reliability, relative reliability enters a self-tuned robust location--scale fit before group normalization, and absolute group reliability attenuates the resulting bounded advantage. The analysis establishes coordinate and second-moment bounds, characterizes exact centering through the weighted location equation, and gives reliability-dependent attenuation guarantees for outlying rewards. We evaluate STAR-GRPO in two complementary reward-hacking regimes. In token-interface exploitation, STAR prevents runaway optimization of the deployed-interface score while improving the canonical quality signal. In rubric-proxy overoptimization for medical reasoning, STAR improves independent semantic evaluation, narrows the proxy--judge discrepancy, and reduces overclaim while optimizing the same task proxy. Together, these results show that reliability-first normalization offers a principled way to limit unsupported reward influence on both group baselines and policy updates, while retaining the task reward as the optimization target.