Human-aligned ai models do not always behave like humans
Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment
Artificial Intelligence
Summary
The paper shows that when AI models are trained to give answers people prefer, their responses can actually become less like what a human would say. This happens because what people prefer from an AI is not always the same as what they would say themselves. The authors call this difference the 'Turing-test gap' and find that existing training methods do not guarantee the model’s behavior matches how humans respond. This means making AI responses human-like needs to be addressed separately from making AI responses align with human preferences.
What this means in practice
- •For ai developers: Design language models that separately optimize for human-likeness and for preference alignment to improve assistant behavior.
- •For user experience teams: Evaluate the human-likeness of AI responses independently from preference metrics to better meet users’ expectations.
Authors
Suqin Yuan, Runqi Lin, Muyang Li, Guanzhe Hong, Jindong Gu, Lei Feng, Chris Russell, Tongliang Liu
Abstract
Human-feedback alignment has made language models useful assistants and is commonly described as aligning them with humans. However, the responses people prefer from an AI need not be the responses they themselves would give. We distinguish alignment with human preferences from alignment with human behavior, and show that alignment with human preferences can make model behavior less human-like even when both preferences and responses come entirely from humans. We call this the Turing-test gap. We show that preference alignment preserves the human response distribution only under a restrictive condition, and find no consistent evidence that real human preferences satisfy it. Empirically, the loss of human-response likelihood increases with the strength of preference weighting, regardless of its direction, and the gap also appears under standard DPO. These results establish human-likeness as an explicit dimension of alignment rather than something assumed to follow from preference alignment.