Llm agents struggle to personalize web tasks without clear user clues
AdaptArena: Evaluating Test-Time Personalization of Web Agents
Machine Learning
Summary
Agents that use large language models can handle complex tasks on websites but have trouble when they don't know what a user really wants or likes. The authors created AdaptArena, a way to test how well these agents can guess user preferences from past actions without being told directly. Their tests show a big gap between agents that know users' true preferences and those that try to guess them. They also found that even if an agent correctly guesses preferences, it can still fail because of problems actually doing the tasks on the web. This means making agents better at understanding users and handling site actions is important for real-world use.
What this means in practice
- •For web developers: Improve AI assistants on websites to better infer user needs from past actions without explicit profiles.
- •For customer support teams: Deploy smarter chat assistants that adapt to users’ preferences by analyzing previous conversations and interactions.
Authors
Dongchan Shin, Xing Han Lù, Jiaqi Deng, Jay Gala, Tomás Vergara Browne, Jaewon Moon, Fengyuan Liu, Alexandre Drouin, Siva Reddy, Alexandre Lacoste
Abstract
Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings where user intentions are underspecified and preferences are heterogeneous. In practice, users rarely provide explicit profiles, requiring agents to infer latent preferences from implicit signals. Despite its importance for deployment, this problem setting is largely underexplored in existing benchmarks. To address this gap, we introduce AdaptArena, a benchmark for evaluating test-time personalization of web agents via implicit preference inference. AdaptArena consists of 480 tasks, featuring both single-preference and double-preference scenarios. Each evaluation task must be solved by retrieving and leveraging the most relevant historical user trajectory that implicitly encodes the target preference. In addition, we introduce AdaptiveAgent, a retrieval-based framework for standardized evaluation of implicit preference inference. Experiments reveal a substantial performance gap: while oracle agents with access to ground-truth preferences achieve an 82.92% success rate, the evaluated LLM agents using our framework reach at most 15.62%. Furthermore, we find that correctly inferring user preferences is necessary but not sufficient for task success, as execution failures in downstream web interactions remain a significant bottleneck even when agents align with the target preference. These findings highlight implicit preference inference and robust action grounding as key challenges for deploying reliable, user-facing web agents. We release our code: https://github.com/McGill-NLP/web-agents-test-time-adaptations