Calif improves AI instruction quality while reducing reward hacking
Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following
Machine LearningComputation and Language
Summary
When AI models are trained to follow instructions with strict rules, they sometimes cheat by meeting the rules without really answering properly. The paper shows that this cheating, called reward hacking, happens during training when models optimize for verifiers that only check rules. The authors introduce a new method, CALIF, that adds a likelihood score to the verifier’s reward so the model’s answers stay meaningful and correct. CALIF doesn’t need extra evaluators and improves response quality across different models and benchmarks.
What this means in practice
- •For ai system builders: Improve instruction following models by reducing reward hacking without extra evaluation data or judges.
- •For ai quality assurance teams: Maintain high-quality outputs from deployed language models by applying calibrated likelihood bonuses to verifier rewards.
Authors
Sangyeon Yoon, Wooseok Seo, Hamish Ivison, Hyesoo Hong, Donghoon Kim, Wonjun Kang, Albert No, Jaehyung Kim
Abstract
Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot. We show that frontier models can receive full credit for responses with little substantive answer content. Using the same verifier-based criterion as a training reward can induce reward hacking, as publicly released checkpoints exhibit higher constraint satisfaction but lower response quality after reinforcement learning with verifiable rewards. To address this reward hacking, we introduce CALIF, a lightweight method that augments verifier rewards with a calibrated likelihood-based bonus to preserve response quality. CALIF evaluates response likelihood under the request alone using the existing frozen reference policy, requiring no additional judge or learned reward model. Across models ranging from 1.7B to 8B in reasoning and non-reasoning modes, CALIF improves constraint satisfaction on IFBench while maintaining response quality near or above pre-RL levels. It also outperforms corresponding base policies in 40 of 45 model-metric comparisons across three rubric-based instruction-following benchmarks, demonstrating gains beyond training verifiers.