Speech decisions made faster by listening without text decoding
Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen LLM Hear Beyond the Transcript
Artificial Intelligence
Summary
Current voice assistants take time to answer many small questions because they convert speech to text before deciding. The authors propose DuplexJev, a system that uses a frozen language model to directly interpret speech signals for quick decisions without needing to decode text. This makes answering questions about spoken input much faster and still accurate, even recognizing speaker traits like gender and emotion. They provide software and data to support this new approach.
What this means in practice
- •For voice assistant developers: Enable voice agents to quickly answer many small questions directly from speech without slow text decoding steps.$Commercial implications: Allows development of faster, more responsive commercial voice assistants with real-time decision-making capabilities.
- •For call center software teams: Add emotion and speaker attribute detection to automated voice systems for better customer interaction understanding.
Authors
Jie Jin, Ziyin Ma, Min Yin, Jinyu Chen, Haigang Song, Zhikun Pang, Xiaowen Zhang
Abstract
Full-duplex voice agents make many small, closed decisions, which current systems answer by slow autoregressive decoding. We propose DuplexJev, which feeds ASR-encoder hidden states through a small connector into a frozen LLM and reads each question as a single-token distribution over its options. Nothing is decoded, and an 8-GPU node answers 80 decisions about eight utterances in about 0.1 s. With a last-layer connector, spoken QA stays close to reading the transcript (90% vs. 91%). DuplexJev also hears the speaker: gender and emotion accuracy both reach 90% (from 55% and 28%) with a cross-attention connector, whose spoken QA drops by only 1 point (83% to 82%). We train decisions with cross-entropy on the read-out answer token, instead of the usual transcript distillation, whose teacher never hears the voice, and keep distillation for content. Encoders and LLMs are interchangeable; we release weights, training recipe, a batched-inference pipeline for full-duplex serving and a bilingual spoken-QA set.