Computer-use agents perform worse when serving diverse user personas
PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas
Human-Computer InteractionArtificial IntelligenceComputers and SocietyMachine Learning
Summary
Computer-use agents usually follow fixed instructions exactly, but in real life, people often give instructions with extra needs or want to approve sensitive actions. The authors created PersonaCUABench, a test that sees how well these agents respond when different types of users rephrase instructions and ask questions in their own way. They found that agents completed fewer tasks correctly when dealing with diverse user personas, and even successful tasks often didn’t fully satisfy the users. This means agent evaluation should consider how well they understand and meet the user's actual needs, not just task completion.
What this means in practice
- •For ai product developers: Improve AI agents to better understand and fulfill varied user instructions across personas by testing with PersonaCUABench dialogue scenarios.$Commercial implications: Enables development of AI assistant products that adapt to diverse user communication styles, improving customer satisfaction and usability.
- •For human-computer interaction teams: Use PersonaCUABench to assess how effectively computer-use agents handle natural user constraints and approvals in simulated conversations.
Authors
Haoming Li, Defu Cao, Xu Hu, Yingtao Luo, Yiyang Wang, Xianghe Xu, Jiashu He, Wanting Jiang, Yifan Zhang, Yunhan Zhao, Siyang Wu, Chen Yang, Haixin Wang, Zheyuan Deng, Taoran Li, Jingwen Zhang, Zhejian Peng, Shushu Liang, Yue Zhao, Bingzhe Li, Feng Chen, Yifan Wang, Jintao Huang, Yijun Wang, Yuexing Hao, Xiaomin Li
Abstract
Computer-use agents (CUAs) are commonly evaluated against fixed task specifications, but users may omit requirements, add constraints, or require approval before consequential actions. Completing the benchmark instruction does not guarantee serving the person who asked. We introduce PersonaCUABench, which holds 74 tasks across 11 domains fixed and varies only who asks. A persona agent, playing one of 44 personas in four groups, rewrites each instruction in that user's own words and answers the CUA's questions, while the CUA sees only the dialogue, never the profile or the original instruction. Across nine backbones (3,330 runs), mean task completion falls by 36% relative to the original requests. The losses have two sources: in 59.9% of cells that lose points, the agent engages with what the user asked for and still gets it wrong, and 11.2% are stops the user requested that the checker scores as failures. Even when a persona request is fully completed, the user gives the top satisfaction rating in only 62.7% of cells, against 84.1% for the original requests. CUAs should therefore be evaluated not only on task completion but also on whether they meet the requirements and constraints users express. Code: https://github.com/MatrAIx-ai/PersonaCUABench. Project page: https://matraix.ai/research/persona-cua.html