Full duplex speech models better handle interruptions and backchannels
Integrate the Reward, Don't Sample It: Window Likelihood for Interruptions and Backchannels in Full-Duplex Speech Models
Computation and LanguageSound
Summary
When computers talk with people, they should let users interrupt but also recognize when users are just giving small signs of listening, like 'uh-huh.' The authors found problems with the usual way of training these talking systems, which sometimes made them stop talking at the wrong times or ignore interruptions. They propose a new method called window likelihood that better teaches the system when to yield or keep talking by looking at the whole decision process rather than sampling just once. This new method improved performance on multiple models and avoided mistaken floor-taking behaviors.
What this means in practice
- •For dialogue system developers: Improve conversational AI to correctly yield during user interruptions and hold for backchannels, enhancing naturalness in voice assistants.
- •For customer service chatbot builders: Create chatbots that better manage speaking turns with customers, reducing awkward silence or interruptions in voice-based support.
Authors
Guoqiang Hu, Lichao Zhang, Yizhou Peng, Yuhao Luo, Haifeng Hu, Sallen Sun, Will Yang, Eng Siong Chng
Abstract
A full-duplex spoken dialogue model must yield when the user cuts in and hold when the user merely backchannels. Sequence-level RL with a timing reward (GRPO) is the usual recipe; we test its whole-rollout form on four open models and find two failures. A binary window reward gives zero advantage on exactly the windows that most need learning (a zero-gradient trap), and the one score per rollout is spread over listening-state decisions the reward is not written in terms of (credit leakage), so the model talks into silence or stops answering. Both come from sampling a reward that is a known function of a few binary control decisions. We propose \emph{window likelihood} (\WL): sample a continuation per window, but integrate in closed form over the stop decisions it visits and maximise the log-probability of acting correctly---a noisy-OR for yielding, a survival product for holding. On HumDial-FDBench, \WL\ gives the best yield/hold F1 on all six structurally different open models we post-train, exceeds Kyutai's released GRPO checkpoints (trained on different data), and adds no re-opening of the floor on any model.