Rlhf performs well when human feedback matches model behavior distribution
Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner
Machine LearningComputer Science and Game Theory
Summary
People train AI language models using human feedback to make them act the way users want, but when different users have different tastes, this can cause problems. The authors studied how a common training method called RLHF can sometimes perform poorly if the preferences given by humans don’t align with what the model expects. They found that this problem isn’t a fundamental flaw in RLHF, and if the feedback data matches the model’s behavior, RLHF works well at balancing different users’ wishes. This means it helps to collect feedback in a way that fits the model’s current behavior or to fine-tune the model first on similar data before training.
What this means in practice
- •For language model developers: Use on-policy preference data or pre-fine-tuning to improve RLHF training and better align models with average user preferences.
- •For recommendation system engineers: Design feedback collection so that preference data closely matches model behavior to reduce utility distortion in multi-user environments.
Authors
Kazusato Oko, Annie Ulichney, Nika Haghtalab, Han Bao
Abstract
While Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning large language models with human preferences, its effectiveness in pluralistic settings has been called into question. Notably, recent work by Gölz et al. (2025) demonstrated that the \textit{distortion} -- defined as the multiplicative gap between the average user utility of the RLHF policy and the optimal average utility -- can scale exponentially with the Bradley-Terry temperature parameter $β$ when users have heterogeneous preferences. In this work, we present a fine-grained analysis of the distortion of RLHF with reward clipping and demonstrate that such exponential degradation is not a fundamental property of the algorithm but rather a consequence of distribution mismatch between the distribution generating preference data ($μ$) and the KL reference policy ($π_{\mathrm{ref}}$). To this end, we establish tight upper and lower bounds on the distortion of RLHF across multiple regimes of the KL regularization strength. We show that in a representative regime, under the Bradley-Terry model, the distortion is $\tildeΘ(βB + β)$, where $B$ is an upper bound on the log density ratio between $μ$ and $π_{\mathrm{ref}}$. In particular, when there is no distribution mismatch (i.e., $μ= π_{\mathrm{ref}}$), RLHF achieves the optimal distortion of $O(β)$ up to a constant. Our results suggest that, to reasonably maximize average utility with RLHF, it is preferable to use on-policy sampled preference data or to fine-tune before RLHF on data from a source close to $μ$.