User simulator evaluation methods miss key behavior errors in interactive agents

On Evaluating User Simulators for Interactive Agents

Machine Learning

Summary

Testing computer programs that talk to users usually uses simulated users to save time, but it's hard to tell if these simulations are really good. The authors checked whether tests for simulated users can spot when the simulated users make big mistakes, like ignoring goals or stopping too soon. They found older tests often miss these errors. The authors made a better testing method that catches all the problems and shows how to verify simulated users match real users closely in how they behave and what they try to do.

What this means in practice

  • For interactive agent developers: Improve evaluation tools to better detect when simulated users fail to behave correctly, leading to more reliable testing of interactive agents.
  • For customer support automation teams: Validate simulated customer sessions to ensure automated support agents are tested with realistic user interactions across languages and tasks.

Authors

Hengguang Zhou, I-Hung Hsu, Li Chen, Priyanshi Shrivastava, Anastasios Angelopoulos, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh

Abstract

User simulators offer a scalable way to evaluate interactive agents, but how to assess their quality remains underexplored. In particular, it is unclear whether existing user simulator evaluation frameworks reliably detect meaningful failures in simulated user behavior. This uncertainty makes it difficult to interpret simulator evaluation. To understand what a high benchmark score tells us about simulator quality, we (meta-)evaluate the metrics themselves. To instantiate this meta-evaluation, we deliberately perturb simulated replays to induce failures such as altered goals, omitted constraints, premature stopping, and acceptance of broken work, then test whether the metrics penalize these failures. Evaluation methods from three prior studies detect only 5-6 of the 10 perturbations; their combined scores even favor populations instructed to accept broken work over unmodified controls. Motivated by these gaps, we propose a new evaluation suite that combines utility of user simulators for evaluating agents and two measures of fidelity to real users: population-level fidelity assesses collective interaction patterns, while trace-level fidelity assesses the preservation of individual users' goals, constraints, and persona. Under the same detection criterion, our evaluation suite detects all 10 perturbations. Evaluating six user simulators on 351 real multi-turn sessions in which 348 distinct users across 23 languages and agents work together to create and refine artifacts such as full-stack web apps, games, design assets, and automated workflows, we find that population-level measures can miss when a simulated user pursues the wrong goal, ignores a constraint, or behaves out of character, highlighting the need to assess fidelity at comprehensive granularities.