EchoChat improves empathetic spoken dialogue with stepwise reasoning
EchoChat: Structured Cognitive Reasoning in Empathetic Spoken Dialogue
Sound
Summary
Having very human-like conversations that truly understand and react to feelings is hard for computers. This work shows that current systems often just sound warm but don’t really grasp the underlying feelings or thoughts of the speaker. The authors created EchoChat, a new method that breaks down empathy into clear steps like recognizing feelings, guessing hidden thoughts, and then replying. They also built a big new dataset and special tools to help computers learn and check these steps better. Their tests show EchoChat can understand and respond with empathy more deeply and accurately.
What this means in practice
- •For voice assistant developers: Build voice agents that understand users’ emotions and mental states step-by-step for more meaningful and supportive conversations.$Commercial implications: Enables voice assistants with deeper empathy features to enhance user satisfaction and engagement, creating commercial chat products.
- •For call center software teams: Improve automated support systems to recognize emotional cues and mental states for better customer care responses.
Authors
Dingdong Wang, Shujie Liu, Yayue Deng, Yuxuan Hu, Yunrui Cai, Jincenzi Wu, Jianwei Yu, Jinyu Li, Helen Meng
Abstract
Empathetic spoken dialogue is a sophisticated cognitive process that requires not only recognizing emotions but also inferring a user's latent mental states to provide appropriate support. However, current SpeechLLMs often treat empathy as a direct input-to-response mapping, leading to "superficially warm" but emotionally hollow interactions. In addition, since empathy relies on a multi-stage process with strong inter-step dependency, errors at any intermediate step can cascade through subsequent steps and lead to inappropriate responses, while existing training paradigms lack mechanisms to precisely localize and improve such errors. In this work, we propose EchoChat, a unified framework that reformulates empathetic spoken dialogue as a structured cognitive reasoning process integrating perception, mental-state reasoning, and response generation. To support this paradigm, we first construct EchoDialogue-400K, an acoustically rich dataset for multi-stage empathetic supervision. During the SFT stage, we strengthen acoustic grounding through proposed Acoustic-Anchored Attention (AAA). During the RL stage, we further introduce a novel stage-aware optimization objective with Step-Decomposed Credit Assignment (SDCA) to localize reasoning errors and mitigate cascaded error propagation. In addition, we introduce EchoEval, an expert-annotated benchmark for multi-dimensional empathy evaluation. Extensive experiments demonstrate that EchoChat achieves state-of-the-art performance in perception, reasoning, and response alignment. Project page: https://github.com/dingdongwang/EchoChat