Mentalized adversarial self play improves proactive dialogue success
Stop Training Against Soft Users: Mentalized Adversarial Self-Play for Proactive Dialogue
Computation and Language
Summary
Proactive dialogue systems talk to users and try to guide conversations toward completing tasks. The authors noticed that many systems are trained against easy or compliant simulated users, which doesn't prepare them for real users who resist or change their minds. They created MASP, a method that treats the conversation as a game between the system and a user who actively resists. This method makes the system better at handling difficult users and finishing tasks faster and more successfully.
What this means in practice
- •For dialogue system developers: Train task-oriented dialogue agents to handle users who resist or change intentions more robustly and efficiently.$Commercial implications: Enables creation of more effective customer support chatbots that can manage challenging conversations and complete tasks reliably.
- •For voice assistant engineers: Build voice assistants that better adapt to users with evolving goals by modeling user mental states during conversation.
Authors
Zanwu Liu, Dan Zhang, Xiaodi Wang, Yongbo Mao, Xiaoyang Guo, Yonglin Wu, Peiyue Jiang, Yangyang Wei, Zecheng Lin
Abstract
Proactive dialogue agents must guide multi-turn interactions toward task goals while adapting to users whose intentions and attitudes evolve. Existing planners improve system side action selection through search, learned policies, or user aware planning, but are often trained and evaluated against fixed or overly compliant simulators; as a result, high simulated success can reflect soft user bias rather than robust dialogue ability. We introduce MASP, a Mentalized Adversarial Self Play framework that treats proactive dialogue as a partially observable game between a system and a rationally resisting user. MASP represents the user's evolving mental state with a BDI based latent state, separates reward side teacher mentalization from policy side student mentalization, and alternates self play updates for the system and user policies. This turns robustness to rational resistance into a training objective rather than only an evaluation condition. Across diverse proactive dialogue benchmarks, compared with the widely used SFT and Planner-only approaches, MASP substantially improves success rate (SR) and rationality constrained success (RCSR), reduces average turns (AT), and performs better against hard users who make the task goal more difficult to achieve.