Understanding limits of preference learning with logistic models
Statistical Learning Theory for Direct Preference Optimization
Machine Learning
Summary
When computers learn what people prefer by comparing pairs, they often assume a simple underlying model of preferences to make predictions. The authors examine what happens when this assumed model doesn’t perfectly fit real preferences, identifying a specific mathematical measure called the triangle-curl to describe the mismatch. They explain how this mismatch limits learning accuracy even if the model is very flexible and more data is added. Their results help separate errors from model choice, structural issues, and limited data.
What this means in practice
- •For machine learning engineers: Build better preference learning algorithms by understanding when logistic models fail to capture real user preferences and how model flexibility affects error.
- •For recommendation system teams: Improve preference-based ranking models by identifying inherent limitations due to structural mismatches beyond data or model size.
A theory result. No direct application yet.
Authors
Joel Wills, Yuanfan Li, Yiming Ying, Ding-Xuan Zhou
Abstract
Direct Preference Optimization (DPO) learns from pairwise preferences through a logistic objective whose score-difference parameterization follows the Bradley--Terry model. We study how failure of this modeling assumption limits DPO, distinguishing structural misspecification from limited model capacity and finite-sample estimation. Our main contribution is a characterization of structural approximation error through the triangle-curl of the preference log-odds, which vanishes exactly when preferences admit a Bradley--Terry representation. We upper bound the achievable excess logistic risk by a score-class approximation term and a triangle-curl term, and establish a lower bound proportional to triangle-curl under additional boundedness assumptions. For ReLU score models, including clipped models, we derive explicit approximation rates showing that increasing capacity can eliminate the score-approximation term while leaving the structural obstruction unchanged. We further establish finite-sample bounds for clipped empirical risk minimizers that separate network approximation, structural misspecification, and estimation error. We place these results within a surrogate-risk framework by identifying contextual preference accuracy (CPA) as a natural measure of agreement between preference judgments and changes in pairwise probability ratios relative to the reference policy. Standard logistic consistency and calibration results connect DPO's objective to this policy property and transfer our excess-risk upper bounds to CPA, without treating CPA as a complete measure of generated-response quality.