ECHO benchmark reveals context matters for turn-taking in dialogue
ECHO: A Matched-Contrast Benchmark for Context-Sensitive Turn-Taking in Full-Duplex Dialogue
Computation and Language
Summary
Spoken dialogue systems that listen and talk at the same time need to know when to stop or keep speaking, especially when people interrupt or give small signals to keep going. The authors show existing tests miss this subtlety by checking events alone rather than with their conversation context. Their new ECHO benchmark uses paired examples of the same overlapping speech but different conversation histories that demand opposite actions. They found many systems tend to stop speaking too often, showing older tests can overestimate how well these systems handle real conversations.
What this means in practice
- •For dialogue system developers: Diagnose and improve turn-taking decisions in conversational AI using paired context-sensitive test cases beyond interruption detection alone.
- •For customer service chatbot teams: Better distinguish when to yield or continue speaking in voice chatbots, reducing awkward interruptions in live agent handovers.
Authors
Shuofeng Zhao, Hongwei Cai, Wenke Fan, Qingxiang Guo, Dawei Yang, Zhou Wang, Zhiyang Zhou, Yingxin Shang, Weixu Wang, Lin Yang, Shuran Zhou, Yang Song
Abstract
Full-duplex spoken dialogue systems must distinguish interruptions that require yielding the floor from backchannels that permit continued speaking. Existing benchmarks typically evaluate events independently and may therefore reward fixed action preferences rather than context-sensitive decisions. We introduce ECHO, a paired diagnostic benchmark for Chinese full-duplex turn-taking. ECHO pairs examples with the same overlap transcript but contrasting preceding multi-turn dialogue contexts, with one requiring Yield and the other Keep. It additionally includes off-talk examples for diagnosing unnecessary yielding. We introduce pair accuracy, which requires correct decisions on both members of a pair and assigns no credit to constant-action policies. Experiments on multiple full-duplex systems show that most exhibit a pronounced bias toward \textsc{Yield}, performing substantially better on interruptions than on backchannels, while another system remains comparatively balanced. These findings demonstrate that interruption-only evaluation can overestimate practical turn-taking reliability. ECHO and its metadata will be publicly released.