UserArena benchmark advances long-term shopping agent simulations
UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents
Information Retrieval
Summary
Shopping recommendation systems try to help buyers by making a series of decisions that guide purchases over time. The authors created UserArena, a large collection of real shopping sessions with lots of details, and a test environment where recommendation methods can be evaluated more realistically over long interactions. They also developed a new training approach called On-Policy Trajectory Distillation to improve how well simulators mimic real user behavior during long shopping sessions. Their experiments show this method best matches actual outcomes recorded in a production test, making simulated training more reliable.
What this means in practice
- •For e-commerce platform developers: Test and improve recommendation agents by simulating long user sessions with verified fidelity to real shopper outcomes.
- •For online marketing teams: Validate new shopping recommendation strategies offline before costly live rollout using UserArena's benchmark and environment.
Authors
Jujia Zhao, Jiakai Tang, Jing Yu, Wen Chen, Jian Wu, Han Zhu, Fei Sun, Suzan Verberne, Zhaochun Ren
Abstract
Agentic recommendation pursues shopper goals through sequences of context-dependent decisions. Developing and evaluating such policies requires faithful recommendation world models that capture both the interactive shopping environment and users' behavioral responses over long interactions. Existing resources rarely provide long-horizon, fine-grained real-user trajectories or support trajectory-level evaluation and offline validation against production A/B results. We introduce UserArena, a real-world e-commerce interactive benchmark containing 64,729 session trajectories from 11,010 users, spanning 566,102 items and 13 action types. UserArena pairs these trajectories with an interactive environment and supports offline validation against outcomes from a controlled production A/B test, enabling five-level evaluation from output validity and local decisions to sequential behavior, personalization, and outcomes. To improve simulator fidelity over these long-horizon interactions, we further propose On-Policy Trajectory Distillation (OPTD), which uses trajectory-level intent to provide dense teacher targets on student-generated rollouts. Experiments show that OPTD achieves the strongest overall fidelity among the evaluated simulators under UserArena's evaluation protocol and most closely reproduces the production policy effect in offline simulation.