Steerable full duplex speech models improve conversational control and timing
SteerDuplex: Steerable Duplex Speech Dialogue Models
Artificial IntelligenceComputation and Language
Summary
People want voice assistants and dialogue systems to talk more naturally and follow instructions about how to speak, like changing tone or speed. The authors found that current full-duplex speech models, which can listen and talk at the same time, were missing this ability to change their speaking style reliably. They created SteerDuplex, a model trained to adjust conversation style and timing based on user instructions, and tested it on a new benchmark called SteerBench. Their model showed big improvements in controlling voice style and handling turn-taking smoothly, though some issues remain with incomplete responses.
What this means in practice
- •For voice assistant developers: Enable voice assistants to follow user instructions to change speaking tone, persona, or speed while managing interruptions smoothly.$Commercial implications: Allows creation of more natural and user-controllable conversational agents for smart devices and customer service.
- •For call center software teams: Improve automated agent responses to handle interruptions and backchannels in real-time while adapting speaking style to better fit conversations.
Authors
Utkarsh Tyagi, Ramaneswaran Selvakumar, Advait Gosai, Sonal Kumar, Nikhil Barhate, Isabell Sagar, Steven Li, Miheer Bavare, Daniel Quigley, Fabiola Tapia Carrillo, Jose M Patron E, Diego Macías Gutiérrez, Paul Song, Ramani Duraiswami, Dinesh Manocha, Yunzhong He
Abstract
Full-duplex spoken dialogue models support low-latency turn taking, interruption handling, and backchanneling, yet a key capability remains underexplored: steerability, the ability to reliably shift conversational behavior along attributes such as tone, persona, speaking rate, and voice style in response to user instructions. We introduce a taxonomy of text- and audio-based steerability that identifies substantial gaps in current full-duplex models. To address this gap, we introduce SteerDuplex, a Moshi-based full-duplex speech model fine-tuned on natural conversations and synthetic dialogues targeting instruction following, vocal delivery, reasoning, and duplex interaction. We further apply two-stage reinforcement learning (RL) with hybrid rewards, combining verifiable interaction checks and judge-based semantic feedback to improve timing and response continuity. To evaluate full-duplex spoken steerability, we introduce SteerBench, a benchmark with 390 spoken prompts and 1,067 human-authored binary audio and text rubrics spanning tone, persona, style/accent, and speed/length. On SteerBench, supervised training improves audio-steering average pass rate by 44.5 percentage points over the strongest evaluated open baseline. On Audio MultiChallenge, task average pass rate improves by 7 points over its strongest evaluated open baseline. RL further raises source-clean interruption response from 72.5% to 82.5% and reduces synthetic pause barge-in from 26.5% to 9%. Steering and aggregate task scores remain comparable or higher, while reward probes reveal reward hacking through incomplete responses. Our model and benchmark support systematic research on spoken steerability, with reward analysis showing why timing gains must be evaluated alongside response completeness.