CompanionBench: A Theory-Anchored, Real-World-Grounded Benchmark for AI Emotional Companionship
2026-08-03 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors created CompanionBench, a new test that better evaluates how large language model (LLM) companions interact with users by using real-world data instead of simulated or scripted conversations. Their benchmark measures ten specific communication skills, like handling unclear situations and responding sensitively, rather than just giving one overall score. They also address biases in judging interactions by using diverse evaluators and statistical methods to fairly compare agents. Testing 28 different agents showed many struggle with genuine emotional support despite appearing friendly. The authors plan to share the test data and code in both Chinese and English for others to use.
LLM companionsbenchmarkinguser simulatorpsychology theoriesItem Response Theoryempathy evaluationsame-family favoritismdisclosure gatecross-lingual evaluationcapability rubric
Authors
Yao Liu, Guangjia Chai, Yuming Huang, Jihao Huang, Lei Wang, Junchen Wan
Abstract
LLM companions are deployed at scale in personally consequential settings, yet poorly evaluated. Existing benchmarks use hand-authored scenarios and prompted simulators, aggregate empathy into one score, and overlook judge biases such as same-family favoritism and scale drift. We introduce CompanionBench, an interactive bilingual benchmark. To our knowledge, it is the first companion benchmark to ground both its scenarios and a trained user simulator in de-identified real-world data. A hidden disclosure gate branches each persona's trajectory on the agent's own behavior, controlling the interaction state space without scripting dialogue. We operationalize ten capabilities derived from 25 theories across psychology and counseling, four of them not graded explicitly by prior work: holding ambiguity, selfobject responsiveness, positive resonance and calibrated challenge. Agents are assessed on two complementary axes: a subjective ten-capability rubric and a deterministic measure of whether deeper disclosure was earned. A cross-family panel dilutes same-family favoritism; an Item Response Theory model separates agent quality from judge severity. Theory fixes what to measure and how personas are structured; real data supply events, history, and profiles -- coverage from theory, authenticity from data. Rankings are reproducible in both languages (rho = 0.996 ZH / 0.953 EN). Evaluating 28 agents reveals capability-level differences obscured by aggregate scores. Emotion regulation and calibrated challenge remain common weaknesses; holding ambiguity discriminates most. Role-play agents rank near the bottom: immersion does not imply relational competence. Across agents, the dominant failure mode is substituting surface warmth for substantive relational support. We will release 500 Chinese-English parallel pairs and the evaluation code.