Groupwise learning reduces unfairness in preference alignment for llms
Groupwise Distortion Guarantees for Preference-Based Alignment
Machine Learning
Summary
Many AI systems learn what people like by comparing pairs of options, but this doesn't always lead to the best overall choices for everyone. The authors build on prior work to help AI systems tailor their responses to different groups of users based on their preferences, improving fairness. They created a new method called GLHF that learns from just one comparison per user and works well for many groups at once. Tests with coffee ratings and AI-generated preferences show GLHF makes group outcomes more fair and better than previous methods.
What this means in practice
- •For ai product developers: Create AI systems that tailor outputs to different user groups improving fairness in satisfaction scores.$Commercial implications: Allows commercial AI platforms to reduce group-level dissatisfaction by serving personalized distributions of responses to users.
- •For customer experience teams: Use group-level preference learning to reduce worst-case dissatisfaction among diverse user segments in recommendation systems.
Authors
Jacob Brodkey, Roberto Tamez, Aaron Roth
Abstract
Preference-based alignment methods such as reinforcement learning from human feedback (RLHF) and Nash learning from human feedback (NLHF) aggregate pairwise preferences to learn an LLM policy, but a natural goal is maximizing social welfare (average cardinal utility), which comparisons alone need not identify. Gölz, Haghtalab, and Yang (GHY) measure the gap by distortion: the worst-case ratio between the welfare of the best fixed lottery (distribution over responses) and of the learned lottery. They show NLHF is optimal when every user receives the same lottery. Account-based LLMs, however, have information about their users and can serve different lotteries to different people. We give an efficient algorithm, GLHF, that learns a single group-conditioned policy from one comparison per user. Under individual Bradley--Terry comparisons, GLHF asymptotically matches GHY's optimal population distortion bound simultaneously on every group in a prespecified, possibly overlapping collection, with sample complexity growing logarithmically in the number of groups and inversely with the smallest group mass. A sharper guarantee for groups with similar preferences approaches distortion of one when members share a feasible favorite response. In experiments using human coffee ratings and synthetic LLM-generated ratings, GLHF lowers distortion in every evaluated group and substantially reduces worst-group distortion relative to NLHF and other group-agnostic baselines.