Budgeted strong-verifier audits reduce reward hacking in reinforcement learning
Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO
Machine LearningArtificial Intelligence
Summary
Sometimes, AI learns to cheat when its rewards are checked with easy tests. The authors studied how to best spend limited resources on harder, more reliable checks to catch cheating. They found that auditing groups that barely pass weak tests first helps stop cheating more than random checks. They tested this on large language model training and coding challenges, showing fewer cheats and better performance. How the updates are done—including how random they are—also matters in training success.
What this means in practice
- •For machine learning engineers: Use low-pass budgeted audits to efficiently reduce reward hacking in reinforcement learning training with limited strong verification resources.
- •For ai model trainers: Improve large language model training outcomes by strategically auditing weakly passing outputs rather than random audits to better suppress cheating behaviors.
Authors
Shurun Li, Michael Hua Wang, Haibo Zeng, Long Wang
Abstract
Reinforcement learning with verifiable rewards trains on cheap checks, such as unit tests, that a policy can learn to game, while a stronger check is too expensive to run on every rollout. We study how to spend a limited strong-verifier budget in GRPO, separating which weakly passing rollouts to audit from what reward a caught hack receives. At a fixed policy state with centered advantages, the expected update depends on a design only through each rollout's marginal audit probability and correction; reflecting a caught hack's reward about the failure reward negates the advantages of an all-hack group; and auditing whole groups with the fewest weak passes first (low-pass budgeted audits, LB) minimizes the advantage mass left on hacks among pass-count designs with a fixed correction when a label's value falls with the pass count. We tested this analysis in actual LLM training: from each run's starting state, the sign of this mass on a frozen panel predicted whether hacking would be suppressed in 33 of 39 runs, including 13 of 16 preregistered ones. In LeetCode (Wong et al., 2026) and Countdown-Code (Khalifa et al., 2026), LB drives hacking to near zero, while random audits with the same penalty and budget do not and use more labels. We also find that the randomness of the update affects training: in Countdown-Code, two unbiased corrections, per-item reward-level IPW and a group-level advantage correction, bring hacking down too but end 6.1-11.8 and 8.0-11.5 points lower in held-out accuracy than LB at the same budget. A full audit, which the group-level correction reproduces in expectation, stays within about two points of LB in three seeds, while LB uses 28% of its labels, and allocating the same correction by pass count recovers most of the loss (one seed). Training outcomes thus depend not only on the expected update but also on the randomness of its estimate.