Wik-talkie benchmarks multilingual chat agents using real user personas
Wiki-Talkie: Multilingual Benchmarking of Persona-Based Agents on Real-World Discussions
Computation and LanguageArtificial Intelligence
Summary
People use AI that talks in social settings, so it’s important to see if these AI really act like humans. The authors created Wiki-Talkie, a new dataset with real conversations from Wikipedia talk pages in five languages, using real user info to make personas. They tested chat agents to see if they could mimic real human replies, finding that the AI do better when they know users' past comments rather than just profiles. However, the AI tend to avoid negative or strong feelings and are more positive and agreeable overall, consistently across languages.
What this means in practice
- •For chatbot developers: Improve chatbot realism by using user comment histories to better simulate interaction behavior across multiple languages.
- •For social media platform teams: Detect and correct overly positive or agreeable biases in automated moderation or engagement agents operating in diverse languages.
Authors
Dennis Fucci, Andrea Bacciu, Dong Liu, Weronika Łajewska, Saab Mansour
Abstract
LLMs are increasingly deployed as autonomous agents in social environments, making it critical to study their ability to faithfully simulate human interactions. Central to this is grounding agents in realistic user personas, yet existing datasets rely on fictional personas and are limited to a handful of languages, lacking the empirical grounding necessary to evaluate behavioral fidelity across diverse populations. We introduce Wiki-Talkie, a multilingual dataset of real-world conversations from Wikipedia Talk pages across five languages spanning two language families: Germanic (German, English) and Romance (Spanish, French, Italian), paired with personas derived from real user communities and encompassing sociodemographic attributes, self-descriptions, and behaviorally grounded interaction traits. Using Wiki-Talkie, we evaluate agent interactional behavior on a next-turn generation task across various persona conditioning strategies. Our evaluation assesses whether agents collectively reproduce the distributional behavioral patterns observed in human discussions. Results show that user's comment history exemplifying interaction behavior consistently outperforms explicit persona information. In addition, models systematically underproduce negative or extreme sentiments, while over producing references and suggestions, revealing biases toward agreeableness and positivity. Crucially, these patterns hold robustly across languages, with small cross-lingual differences.