How stated preferences differ from what people reward in ai responses
Abstract Priorities, Concrete Rewards: How Stated Preferences Relate to What People Reward in AI Responses
Human-Computer InteractionArtificial IntelligenceComputation and Language
Summary
People often say what they want from AI answers, like shorter responses or clearer language, but what they actually reward when rating these answers doesn't always match. The authors studied over a thousand people and found that while people’s stated priorities matched what they thought about AI responses, these priorities didn't predict the specific features they rewarded. For example, someone who asked for shorter answers might not give higher ratings to shorter responses. This means that what people say they want and how they actually rate AI responses can be quite different.
What this means in practice
- •For ai product designers: Tailor AI response evaluations by focusing on perceived qualities rather than just stated user preferences for better personalization.
- •For user experience teams: Improve feedback collection by distinguishing between what users say they want and what features they actually reward in AI responses.
Tested on one dataset.
Authors
Zihan Chen, Lei Nico Zheng, Di Zhu
Abstract
AI assistants built on large language models (LLMs) are trained on human feedback and increasingly personalized to users' stated preferences. Yet how closely these preferences reflect what people reward in responses remains unclear. We compare stated and revealed preferences in the public PRISM dataset, in which 1,396 participants reported how much seven response attributes mattered to them and then rated 29,347 responses from 21 LLMs. Among responses to the same prompt, features the survey never asked about, such as response length, formatting and refusal language, were at least as strongly associated with ratings as text proxies of the surveyed attributes. Whether stated priorities matched ratings depended on how attributes were measured: participants who prioritized an attribute favoured responses from models that others perceived as strong on it, but not responses that scored higher on text proxies for it. Written feedback was more telling: participants who asked for shorter answers, even in earlier conversations, gave length less weight in their ratings, and those asking for longer answers gave it more. Their relative priorities over the surveyed attributes, however, did not predict this weight. Reasons given for choices reflected perceived attributes more than measured features. Participants also differed stably in how much they weighted several concrete features, especially length, but with a median of 6 conversations per participant, these weights were estimated poorly. The largest gain from personal information in predicting held-out choices was 0.13 percentage points, compared with 1.11 for a stronger pooled model. Stated priorities thus track perceived attributes of AI responses but say little about the concrete features that ratings reward, a distinction that matters for preference elicitation and AI personalization.