SocialRL improves multi-turn dialogue for smarter AI conversations
SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design
Computation and Language
Summary
Having a good conversation with AI means understanding both the goal and the relationship between people in the talk. The authors found that current AI chat methods only focus on one message at a time, which misses the bigger picture over many turns. They created SocialRL, which teaches AI to think about the whole conversation, balancing getting things done with being friendly and clear. Their approach improved AI’s success in several social chat tests by about 9%.
What this means in practice
- •For customer service teams: Enhance AI chatbots to handle longer conversations balancing task success and customer rapport for better service outcomes.
- •For virtual assistant developers: Build virtual assistants that adapt their responses over multiple dialogue turns to maintain user engagement and achieve goals.
Authors
Jianing Wang, Xintao Wang, Aili Chen, Jie Shi, Hongcheng Guo, Jun Gao, Wenxuan Zhao, Chengkun Lang, Yuanli Guo, Yanghua Xiao
Abstract
Social intelligence enables agents to read social context, infer intent, and adapt over sustained dialogue. As language models become autonomous collaborators, it is central to building effective and trustworthy human-AI interaction. Existing reinforcement learning methods optimize single-turn utterances and sparse outcome rewards, producing short-sighted policies that struggle to manage goal-relationship tensions across multi-turn interactions. We propose SocialRL, a multi-turn reinforcement learning framework addressing both challenges. First, we apply multi-turn reinforcement learning using PPO that propagates delayed outcome rewards back to each turn, enabling long-horizon planning. Second, we design six process reward dimensions capturing the goal-relationship trade-off, including goal advancement, relational attunement, contextual coherence, etc. A reward model dynamically generates fine-grained scoring criteria for each dimension, while a stage-aware weight schedule prioritizes relationship-building in early turns, goal advancement mid-way, and balanced closure late. Across multiple social-dialogue benchmarks, SocialRL improves Goal Achievement by an average of 9.2 percentage points over the corresponding Base models. These results demonstrate the effectiveness of SocialRL across synthetic and real social scenes, as well as standard and challenging social scenarios.