Socialarena measures social skills of large language models
SocialArena: Evaluating Social Intelligence in Large Language Models
Artificial Intelligence
Summary
Large language models are being used in sensitive social roles like counseling and tutoring, but current tests don’t fully check their social understanding. The authors introduce SocialArena, which tests models on role-playing, understanding others’ thoughts, and predicting behaviors in many social situations. They also hired trained human judges to carefully compare model answers and provide detailed feedback. This helps make the evaluation more reliable and closer to real human interactions.
What this means in practice
- •For chatbot developers: Evaluate and improve social skills of chatbots interacting with users in counseling or tutoring contexts.
- •For human resources teams: Assess AI-based virtual assistants for realistic role-playing and social reasoning in candidate screening and onboarding.
Authors
Zongqi Wang, Rui Wang, Yuchuan Wu, Hua Yang, Bingxue Qiu, Tianle Gu, Bing Zhao, Yujiu Yang, Jieping Ye, Yongbin Li
Abstract
We introduce SocialArena, a benchmark for systematically defining and evaluating the social intelligence of large language models (LLMs). Although LLMs are increasingly deployed in socially interactive applications such as mental health counseling, companionship chatbots, and educational tutoring, existing evaluations remain fragmented focusing on isolated capabilities or single scenarios and evaluation reliability is largely unverified. This work makes three main contributions to address these limitations. (1) We first propose a unified three-tier framework that defines social intelligence along three capability dimensions: role-playing ability, mental state reasoning, and behavioral reasoning. To operationalize this definition, we design a role-playing dialogue paradigm with explicit character thinking for evaluation, enabling simultaneous assessment of these capabilities in a manner that mirrors realistic human interactions. (2) To ensure diversity across social scenarios and thereby prevent evaluation conclusions from being confined to specific scenarios, we introduce the U-TRIMF framework, which guides us in synthesizing social contexts with broad coverage. (3) To obtain reliable evaluation, we invest $132.4K to recruit and train human annotators and adopt a human-annotated arena-style protocol, where annotators compare model responses and provide preference labels along with natural-language critiques. We hope that SocialArena can provide valuable insights and guidance for future LLM improvements.