Persona vectors improve user input realism in AI agent tests
A Three-Tier Persona Vector for Controllable User Simulation in Agentic Evaluation
Artificial IntelligenceComputation and Language
Summary
Testing AI assistants that use large language models requires them to talk to different kinds of users. The authors designed a way to create detailed user profiles with many traits like age, patience, and emotions. These profiles change depending on the situation, producing varied and realistic conversations. When tested, these detailed user profiles led to noticeably different outcomes for the AI agents, showing they can better mimic real human behaviors.
What this means in practice
- •For ai product testers: Use persona vectors to create diverse, realistic user inputs for more accurate AI assistant performance testing.
- •For customer service bot developers: Design bots that better handle varied customer behaviors by training on scenario-reactive personas with emotional and behavioral traits.
Authors
Rahul Khedar, Eshita, Sneha Teja Sree Reddy Thondapu, Mayank Malhotra, Arup Kumar Das, Jitesh Chandra Mishra, Arun Menon, Avinash Karn, Mouli V
Abstract
Evaluating tool-augmented LLM agents requires diverse, realistic user inputs yet most evaluation frameworks use flat role descriptions ("you are an angry customer") that produce near-identical conversations regardless of the underlying scenario. In this paper, we propose a three-tier persona vector with 23 operationalized dimensions: 6 categorical demographics (jurisdiction, age, channel, device, language proficiency, time availability), 12 continuous behavioral traits (patience, assertiveness, digital literacy, etc.) sampled with Gaussian noise around curated profile base vectors, and 5 continuous emotional states (frustration, anxiety, trust, confidence, stress) that shift in response to scenario context. Orthogonal to the persona, a 4-level query-complexity overlay controls utterance phrasing from direct to deliberately vague. We evaluate the persona model inside a synthetic data generation pipeline across 64,698 multi-turn conversations spanning 8 named profiles and 3 production corpora. Key findings: (i) a 15.8 percentage-point spread in agent goal-achievement across personas confirms trait vectors produce measurably different user behavior; (ii) the same persona behaves differently across scenarios due to scenario-reactive emotional state shifts, validating the scenario-reactive design; (iii) domain-specific projects show persona sensitivity on booking-flow compliance (~15-20 percentage points gap between tier-aware and pressure-test personas), demonstrating the model faithfully reproduces real-world difficulty distributions; (iv) seven rule-described trait correlations produce auditable co-occurrence patterns without requiring learned covariance matrices. The persona model is fully specified for reproduction.