Six speech models tested on how naturally they join conversations
People Making Conversation: Evaluating How Naturally Speech Models Converse
Computation and Language
Summary
People expect speech models to know when to speak during a conversation, but most current systems always talk at pauses without knowing whether they should. The authors tested six different speech models using real conversations to see if they speak at natural times, how quickly they respond, and if they can remember details from the discussion. They found that only some models stay silent when appropriate and interrupt like humans do. Also, models with access to text remembered more facts than those relying only on audio.
What this means in practice
- •For voice assistant developers: Evaluate and improve when voice assistants speak during natural conversations using the detailed pause and turn-end analysis method.
- •For call center technology teams: Use conversation scoring techniques to better detect appropriate speaking times and improve agent-to-caller dialogue flow in real interactions.
Authors
Arya Bastani, Iman Modarressi
Abstract
A conversational speech model has to decide, at every pause, whether to speak. Existing evaluations test boundary detection, policy adherence, or aggregate statistics, not this decision. We place real-time speech systems at 427 pauses and 482 turn ends from 176 unscripted conversations in Seamless Interaction, give each the conversation so far, and score what it generates against what the person did: whether and how soon it speaks at a pause; its latency, length, and wording at a turn end; and whether it can answer a question about the preceding 30 to 120 seconds. Across six systems, the hosted APIs (GPT Realtime, Gemini 3.1 Flash Live, Grok Voice) respond at almost every pause whichever endpointer they use; only their latency, about 1 to 4 s with silence-based endpointers and about 8 s with semantic turn detection, decides whether the words land inside the pause. The full-duplex models (PersonaPlex, Moshi) are the only systems that regularly stay silent and the only ones that interrupt within a human reaction time. Hosted systems reply longer than the person. Given the history as text, hosted systems recall 41 to 60% of conversation facts under a Claude Opus 5.5 judge and 19 to 26% under a strict word matcher; hearing it as audio, full-duplex models recall 17 to 22% and 12 to 15%. We release the labels, points, harness, and every output.