Full duplex voice agents adapt speech while others talk
Continue, Adapt, or Yield: In-Turn Adaptation to Overlapping Speech in Full-Duplex Agents
Computation and LanguageSound
Summary
Talking with a voice-controlled assistant can be tricky when people speak at the same time. The authors point out that humans don’t just stop or continue speaking when interrupted; they often adjust their words on the fly to include what the other person said. They developed a new way, called Duplex Cue, to measure how well AI voice agents do this kind of in-turn adaptation. Testing with a model named PersonaPlex showed it adapts less than humans, suggesting current AI still struggles with natural back-and-forth talking.
What this means in practice
- •For voice assistant developers: Improve AI agents to better adjust speech mid-turn when users overlap speaking with corrections or clarifications.
- •For customer support centers: Enhance automated agents so they handle overlapping conversations more naturally by adapting instead of abruptly stopping or ignoring interruptions.
Tested on one dataset.
Authors
Yunqi Lu, Tyler Baumgartner, Nikhil Johri, Brandon Tai, Candice Fan, Luc Debaupte, Ruben Aguilar, Bill Wang, Yi Zhong
Abstract
Full-duplex evaluation often emphasizes whether an agent keeps speaking or stops. That binary cannot express a third response humans use routinely: continuing to speak while incorporating what the listener just contributed. The contribution may be a missing word, a correction or a clarification. We introduce Duplex Cue, an evaluation of this \emph{in-turn adaptation} in full-duplex voice agents. Duplex Cue separates listener intent (backchannel, collaboration, or interruption) from speaker behavior: continuing unchanged, adapting within the turn, or yielding. Adaptation includes acknowledgment as well as content revision. In a single-model case study using 300 human-confirmed cues from unscripted English conversations, we compare recorded human responses with PersonaPlex continuations generated while replaying the listener's audio. We retain 208 pairs with the ongoing speaker active at cue onset and a scorable response in each condition. On the 66 collaborative pairs, recorded speakers adapt in 68.2\% of cases, compared with 34.8\% for PersonaPlex. The model otherwise continues unchanged (42.4\%) or yields (22.7\%). These findings show why evaluating natural voice interaction requires measuring how an agent responds to a listener's contribution as well as whether it keeps speaking.