Social robots learn dialogue strategies from both success and failure
Learning from Success and Failure: Acquiring Adaptive Dialogue Strategies for Social Robots
Human-Computer Interaction
Summary
Social robots need good ways to talk to people and understand their traits, but collecting data for this is hard and often includes lots of failed conversations. The authors developed a method that uses smart image-and-language and language models to recognize user traits and learn how to respond better by looking at both successful and unsuccessful interactions. They found that including what didn’t work helps the robot improve its conversations. This approach could make it less expensive and easier to build robots that adapt well to different people.
What this means in practice
- •For social robot developers: Create adaptive dialogue systems that improve by learning from both failed and successful interactions with users.
- •For customer service chatbot teams: Enhance chatbots by integrating user attribute recognition and failed interaction analysis to tailor dialogue strategies.
Authors
Sanae Yamashita, Yuki Okafuji
Abstract
Traditional dialogue systems for social robots require both dialogue strategies and user attribute recognition, each demanding specialized expertise. However, data collection is costly in real-world deployments, and the resulting datasets often include many failure cases. In this study, we aim to automate the acquisition of dialogue strategies by leveraging both successful and failed interactions using a vision-language model (VLM) and a large language model (LLM). We propose an architecture in which user attributes, recognized by the VLM, along with dialogue history, are fed into the LLM to generate dialogue strategies tailored to specific user attributes. We extracted dialogue strategies from an interaction dataset collected through a field experiment and evaluated their effectiveness. The results demonstrate that explicitly representing failure strategies complements success strategies and improves performance. Our findings highlight a practical pipeline for constructing and maintaining an interpretable strategy repository from in-the-wild deployment logs by recycling abundant failure interactions as reusable constraints, ultimately reducing the development cost of social robots.