Calibrated user embeddings improve multi-turn ai agent benchmarking
CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking
Computation and Language
Summary
Evaluating AI agents that interact with people multiple times usually uses simulated users to test how well the AI works. The authors find that current simulated users do not accurately reflect when or why the AI succeeds or fails compared to real people. They introduce a new technique called Calibrated User Embeddings (CUE) that captures real user behavior and uses it to guide AI simulators more realistically. This approach makes simulations better at matching real user results across different tasks without needing extra training.
What this means in practice
- •For customer support teams: Assess AI chat agents more accurately by simulating user behavior that better matches real customer interactions and failure patterns.
- •For educational software developers: Improve evaluation of tutoring AI by simulating realistic student personas that reflect actual learning successes and failures.
Authors
Anjali Kantharuban, Jonas Mueller
Abstract
Recent benchmarks rely on user simulators to evaluate AI agents in multi-turn interaction. While existing simulation techniques demonstrate surface fidelity to human style and behavior, ecologically valid interactive benchmarking also requires alignment in when and how agents fail across simulated and real user populations. We find that existing simulators lack outcome calibration: agreement with observed success rates and failure patterns when real users interact with the same agent. We introduce Calibrated User Embeddings (CUE), a framework that both encodes observed sessions and samples continuous representations, then decodes them into persona commands to steer LLMs to act as user simulators without training. Through this, we evaluate user-conditioned replay of past sessions and aggregate metric agreement when sampling novel personas for the same tasks. On $τ^2$-Bench, CUEd simulators commit fewer simulator-attributed errors and more faithfully reproduce real-user agent failure modes, aggregate success rates, and outcomes for specific task-user pairs than other persona-based simulation methods. These gains coexist with competitive user fidelity as measured using metrics established in prior work. After being fit to mostly customer support interactions, the same CUEd simulators generalize to document creation, math tutoring, and casual conversation, and remain effective across different simulator LLMs without CUE retraining.