Policies find errors in human preference models using strategic games
Can You Trust Your Preference Models? Stackelberg Games for Alignment
Artificial IntelligenceComputer Science and Game Theory
Summary
Human preference models used in AI can sometimes make mistakes that get worse during training. The authors developed a way to find cases where these models might reverse what humans actually prefer by having two policies compete in a game. One policy tries to hide disagreements, while the other tries to increase them, helping to find where the models are likely wrong. This method was tested with simulated data and showed it can find these tricky cases better than some previous approaches.
What this means in practice
- •For ai alignment engineers: Improve identification of errors in preference models to guide additional human feedback during AI training.
- •For machine learning model developers: Train policies that generate comparison tasks highlighting possible mistakes in evaluator models to improve model reliability.
Tested on simulated data.
Authors
Riccardo Formenti, Aditya Agrawal, Vikas Garg
Abstract
Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons on which an evaluator may reverse the underlying preference, providing candidates for additional supervision. Since human preferences are unavailable during policy training, we use disagreement among an ensemble of preference models as a proxy for prediction error. We formulate this search as a zero-sum Stackelberg game between two opposing policies, a Leader which tries to minimize disagreement, and a Follower which maximizes it. For a general class of finite contextual Stackelberg games, we establish equilibrium existence and, under regularization, uniqueness and a closed-form characterization. We introduce Stackelberg Mirror Descent-Ascent (SMIDA), a two-timescale algorithm with linear last-iterate convergence in the tabular setting, and develop a neural policy-gradient analogue. Experiments with a synthetic oracle support disagreement as an informative proxy for prediction error. On held out data the trained policies outperform reference policies and increase the fraction of comparisons that exceed a disagreement threshold and reverse the oracle's preference.