Full-duplex speech models struggle to maintain long conversations
Why Full-Duplex Speech Models Collapse in Long Conversations
Computation and LanguageSound
Summary
Full-duplex speech models talk and listen simultaneously, making conversations more natural. But these models have trouble holding a long conversation because their memory fills up quickly and their responses start to degrade. The authors found that models either forget early conversation details or repeat, stay silent, or produce nonsense. They identified specific reasons why this happens and introduced two fixes that help models talk meaningfully for up to an hour without retraining.
What this means in practice
- •For voice assistant developers: Improve voice assistants to hold longer natural conversations without losing context or repeating themselves.
- •For customer service chatbot teams: Enhance chatbots to reduce repetitive or nonsensical replies during extended customer interactions.
Authors
Yeonju Kim, Bella Godiva, Se Jin Park, Yong Man Ro
Abstract
Full-duplex speech models listen and speak at the same time, which makes them promising for natural, open-ended conversation. But can they sustain a coherent conversation for tens of minutes? Because they generate a token every frame, even while listening, they exhaust their context windows within minutes. Yet existing benchmarks evaluate only single turns or a few conversational rounds, too short to expose the long-horizon problem. We introduce the Full-Duplex Long-Horizon Benchmark (FD-LHB), which tests five recent full-duplex models on long replayed conversations and in live conversation with an LLM partner. Every evaluated model eventually either degrades or exhausts its context. Models that keep the full history often degrade before their context runs out, while sliding-window models still fall into repetition, silence, and gibberish. For sliding-window models, we identify two contributing mechanisms. Gibberish and silence are linked to Initialization-Anchor Eviction, where the sliding window discards the first KV entry. Repetition is also reinforced by the model's own recent outputs. Guided by this diagnosis, we apply two simple fixes: Initialization-Anchor Pinning (IAP) keeps that first entry, and Duplex Repetition Penalty (DRP) discourages echoing while guarding against prolonged silence. Together, they support the diagnosis and, without retraining, substantially reduce conversational collapse in replays of up to an hour while running in real time.