Summary
Sometimes improving a score used to evaluate language models doesn't mean the model is actually better at its task. The authors look at how optimizing the model in different ways—by changing its internal settings, selecting the best outputs, or editing persistent prompts—can exploit flaws in the evaluator, leading to misleading progress. They provide a framework to compare these different optimization methods and discuss how defenses against these tricks might work or fail across methods. This helps people understand when improvements are trustworthy and how to check real task performance beyond just scores.
What this means in practice
- •For software developers: Assess evaluation methods to avoid false progress when improving language model behavior across different optimization approaches.
- •For ai product managers: Design verification strategies that maintain real task quality independent of measurable scores to ensure reliable AI improvements.
A theory result. No direct application yet.
Abstract
A higher evaluation score does not always mean a better language model system. When optimization exploits an evaluator's mistakes, measured progress can conceal unchanged or deteriorating task performance. This failure can arise through parameter updates, selection among generated outputs, or revisions to persistent prompts. We develop a comparative framework for reward hacking across these three optimization substrates: weights, selection, and text. Building on the Proxy Compression Hypothesis and research on inference-time and in-context reward hacking, we examine how reachable behavior, optimization budgets, and persistent adaptation shape exposure to proxy error. We formalize a distance-dependent upper bound on evaluator disagreement and a capacity ordering for nested policy classes, then show why distance alone cannot establish a universal ranking of vulnerability. An exact finite-output illustration demonstrates how the location of a scoring defect changes the behavior favored by each method. We also map representative defenses across substrates, identifying which mechanisms transfer directly and which offer only functional analogies. Persistent prompts receive particular attention: their contents are inspectable, but the behavior induced by a small textual change may be difficult to anticipate. The formal analysis, numerical illustration, and published evidence together provide a basis for comparing optimization methods and identifying the conditions under which their defenses transfer. The resulting framework connects optimization choices to verification requirements: reliable improvement depends on controlling accessible failure modes and preserving evidence of task quality independent of the score being optimized.