LLM agents learn to adapt socially using persona-driven simulations

PB-GRPO: Learning Socially Adaptive LLM Agents from Persona-Driven Simulation with Preference-Batched GRPO

Computation and LanguageArtificial Intelligence

Summary

Getting AI chatbots to behave well socially means they need to understand unspoken goals, respect preferences, and adapt during conversations, which is tricky because real data is limited and user likes are often hidden. The authors created a simulated environment where AI agents talk to virtual users with different social personas and get feedback on how satisfied the users are. To teach agents better social behavior, they developed a new algorithm called PB-GRPO that improves learning by grouping similar user preferences together during training. Tests showed this approach leads to more socially adaptive AI chatbots compared to earlier methods.

What this means in practice

  • For chatbot developers: Train chatbots to better adapt to users’ social preferences by leveraging persona-based simulations and improved reinforcement learning techniques.
  • For customer service teams: Deploy AI agents capable of interpreting and responding to diverse customer preferences over multiple interactions to improve satisfaction in automated support.

Authors

Jingquan Wang, Jun Yin, Xu Han, Yongsheng Mei, Jie Hao, Bin Guo

Abstract

Building LLMs that behave well socially, not merely correctly, requires Building LLMs that behave well socially, not merely correctly, requires more than producing locally helpful responses. A socially competent agent must infer users' unstated goals, respect their preferences, and adapt as the conversation unfolds. These behaviors are inherently multi-turn and social, making them hard to optimize: real interaction data is scarce, and user preferences are typically latent rather than directly observable. To address these challenges, we build on a persona-driven social simulation environment (consisting of a persona library, LLM-based user simulators, and a user-satisfaction scoring system ranging from [0, 1]), to introduce preference-batched GRPO (PB-GRPO), a post-training algorithm that learns socially adaptive policies from conversation-level feedback. Compared to vanilla GRPO, PB-GRPO computes advantages using a normalization estimated across a bucket of users with similar preferences, stabilizing training across a diverse social population. Empirical evidence shows that PB-GRPO improves models' social behavior over strong reinforcement learning baselines in our simulated environment.
View PDFOpen arXiv