VoxTurn improves voice agent turn-taking decisions with one model
VoxTurn: Answering Several Turn-Taking Questions in One Pass with One Model
Sound
Summary
Voice assistants must decide when users have finished speaking to respond quickly without interruptions. Current methods use multiple models for different related decisions, causing mistakes and delays. The authors present VoxTurn, a single large speech model that answers several turn-taking questions all at once. It performs well across many languages and is faster and more accurate than open alternatives, cutting user interruptions by a notable margin.
What this means in practice
- •For voice assistant developers: Use VoxTurn to improve how voice agents detect when users finish speaking, reducing cut-offs and latency across multiple languages.
- •For speech platform engineers: Integrate VoxTurn into conversational systems to consolidate several speech turn-taking decisions into a fast, single-pass model call.
- •For call center technology providers: Build more responsive voice-based customer service tools by adopting VoxTurn’s approach to end-of-turn detection and related speech event classification.$Commercial implications: Enables improved voice agent responsiveness and user experience, which can be sold as a feature to enterprise call center software providers.
Authors
Yang Xiao
Abstract
A cascaded voice agent must decide, throughout each silence of its user, whether the user has finished their turn: answering too early cuts the user off, answering too late makes the agent slow. It often needs related decisions too: will the user soon go on, is speech during its own turn an interruption or a backchannel, is a sound speech at all. With open detectors, each extra decision costs another model or another call, and on the 14-language eot-bench they cut users off in more than a quarter of mid-turn pauses. Because they are trained on segmented utterances, this mismatch is the largest single source of error. We present VoxTurn, a 0.8B speech LLM that casts these decisions as multiple-choice questions, scores the hidden state of each option, and answers any set of them in one forward pass with a block-isolating attention mask, exactly as if asked one at a time. Trained on real conversational pauses with labels derived from their timing, and re-queried while a silence lasts, VoxTurn reaches 11.5% false cut-offs in English at a 300 ms latency budget, comparable to two commercial detectors, and a 14-language mean of 19.9%, the lowest among open models, at 8.3 ms per decision. In Pipecat and LiveKit it cuts off 17% and 9% of turns, and it ranks fifth of 24 in TurnBench end-of-turn detection without any TurnBench training data. The model, code and framework plugins are released.