LLM user simulators improved by reducing trait mixing in preferences
A Behavioral Trait Leaks into Preferences: Diagnosing Trait Interference in LLM User Simulators
Artificial Intelligence
Summary
Evaluating recommendation systems often uses AI models to simulate how users behave and what they like. The authors found that when these models try to separately control how much users browse and what they prefer, the behaviors mix up incorrectly, leading to misleading evaluations. They propose a new method called PQA that helps the simulator decide when a user would stop viewing irrelevant content, keeping browsing and preferences properly separate. This fix makes testing recommendation systems more reliable under different browsing behaviors.
What this means in practice
- •For recommender system engineers: Better evaluation of recommenders by avoiding bias from mixed user behavior traits in simulated testing environments.
- •For e-commerce platform designers: Enhance user engagement modeling by accurately simulating how browsing activity relates to user preferences for targeted content delivery.
Authors
Chaehyun Kim, Sein Kim, Hongseok Kang, Chanyoung Park
Abstract
LLM-based user simulators aim to bridge the offline-online gap in recommender evaluation by emulating users through injected traits, where preference attributes determine what a user engages with and a behavioral activity trait governs how long they browse. However, we show this intended trait independence collapses during simulation, causing two failures: (i) Trait Interference, where amplified activity distorts preference boundaries and forces interactions with mismatched items to sustain browsing, and (ii) Evaluation Invalidity, where satisfaction scores inflate with activity-driven page counts despite taste mismatches, biasing evaluation toward trait distributions rather than recommender performance. To resolve this, we propose PQA, a page-level quality anchoring method that guides simulators using a personalized anchor reflecting each user's intrinsic preference standard. By assessing whether a page meets this standard before further browsing, PQA enables proactive exits from low-quality pages, letting the activity trait retain its intended role of modulating browsing depth within preference-conforming pages. Experiments show PQA mitigates trait interference and improves the reliability of LLM-based simulator evaluation under activity shifts. Our code is available at https://github.com/chaehyun1/PQA