Speech dialog models speed up with rolling prediction technique
DiffuPlex: Accelerating Full-Duplex Spoken Dialog Models via Rolling Masked Diffusion
SoundComputation and Language
Summary
Talking devices that listen and speak at the same time usually update what they say very slowly, one small part at a time. The authors introduce a new method called DiffuPlex that predicts several parts ahead all at once, speeding up the process. It checks these predictions as the person talks and only changes future parts if needed, keeping past speech stable. This approach makes conversations happen faster without losing much quality, though sometimes the speech sounds a little less natural.
What this means in practice
- •For voice assistant developers: Reduce processing delays in simultaneous listening and speaking for more responsive assistants by predicting multiple future speech frames at once.
- •For call center technology teams: Improve real-time automated agent response time during customer calls by cutting down sequential speech model computations while keeping conversation quality.
Authors
Heeseung Kim
Abstract
Recent full-duplex spoken dialog models enable simultaneous listening and speaking, but fine-grained models still advance their backbone autoregressively at every interaction frame. We introduce DiffuPlex, a rolling masked diffusion framework that reduces this sequential computation by predicting multiple future user and assistant frames in a single backbone wake. DiffuPlex consumes only a confident prefix of each predicted future while interaction continues at the original frame rate. As user speech arrives, it checks the corresponding user predictions and, when the interaction diverges, preserves already played assistant content while revising only the unplayed future. We consider two inference policies over the same predictor: DiffuPlex-LISTEN consumes multiple future frames when they predict assistant silence, whereas DiffuPlex-SPEAK can also consume predicted assistant speech. Across full-duplex interaction and spoken-language evaluations, DiffuPlex substantially reduces sequential backbone computation while largely preserving interaction behavior and general capability. DiffuPlex-LISTEN and DiffuPlex-SPEAK achieve $1.46\times$ and $1.59\times$ deployment-path wall-clock speedups and $1.61\times$ and $1.80\times$ Core LM speedups, with all measured backbone invocations completing within the 80ms interaction interval. Human evaluation shows that LISTEN preserves speech naturalness and conversational quality, while SPEAK retains conversational quality with some degradation in speech naturalness.