UXBench: Benchmarking User Experience in AI Assistants
2026-06-08 • Computation and Language
Computation and LanguageHuman-Computer Interaction
AI summaryⓘ
The authors created UXBench, a new way to test AI assistants based on real user feedback, focusing on how well these systems match user preferences and handle conversations. They collected a large dataset from a popular Chinese AI assistant, covering many scenarios and types of errors users face. By testing 26 language models, they found that models can learn to predict user satisfaction and revealed some biases in how current evaluation methods work. Their work highlights the need to improve AI assistants by focusing more on actual user experience.
AI assistantUser experience (UX)Preference alignmentDialogue generationBenchmarkLanguage modelReward modelEvaluation protocolUser feedbackBias
Authors
Mengze Hong, Xia Zeng, Zeyang Lei, Sheng Wang, Chen Jason Zhang, Di Jiang, Taiming Fu, Jinfeng Huang, Mengqiao Liu, Qinghe Chang, Haosheng Zou, Qiongyi Zhou, Sijun He, Chen Xiaoshuai, Simon Deng, Haojing Huang, Zijian Li, Lucas Mu Li, Fubao Zhang, Mona Zhou, Wei Ma, Chenxuan Ma, Yuanmeng Zhang, Jian Song, Minlong Peng, Di Liang, Davey Chen
Abstract
As AI assistants serve millions of users daily, evaluating user experience (UX) beyond general model capability has become increasingly important. We present UXBench, the first user-centric benchmark grounded in real user feedback signals for evaluating preference alignment and dialogue generation. The benchmark consists of three interconnected tasks, UX Judge, UX Eval, and UX Recovery, with 7,400 test instances extracted from over 70K interaction logs of a mainstream Chinese AI assistant. The dataset closely reflects real user distributions, covering 8 scenarios, 83 domains, and diverse failure patterns that pose severe challenges. Extensive experiments on 26 frontier language models provide novel insights into how well models perceive user experience and how improvements in model capability contribute to better dialogue engagement. Through comprehensive analysis of model behavior and performance gaps, we show that user feedback prediction is a learnable capability, where a reward model trained from in-the-wild feedback signals can achieve well-calibrated accuracy. We further document the systematic biases of LLM-as-a-judge evaluation protocols and compare typical response strategies that directly affect user experience. UXBench establishes a new evaluation landscape and calls for greater attention to tailored UX optimization, contributing to a user-centric scaling law that shapes the success of AI assistants.