TASTE2 enables real-time full-duplex voice interaction with interruption handling
TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction
Sound
Summary
Having smooth back-and-forth voice conversations with computers is hard because it requires the system to listen and talk at the same time while understanding speech and responding quickly. The authors tested a method called TASTE2 that turns speech into shared text tokens to better handle interruptions and keep conversations flowing naturally. They built a voice robot that can listen and respond in real time, even if the user talks over it. While it works well, there is still room to improve the speed and naturalness of the generated speech.
What this means in practice
- •For voice assistant developers: Create voice assistants that listen and speak simultaneously while managing interruptions smoothly.$Commercial implications: Enables development of commercial voice agents with natural, real-time full-duplex conversation capabilities.
- •For interactive voice response teams: Build dialogue systems that process streaming user speech and generate responses incrementally to reduce wait times.
Authors
Yi-Chang Chen, Chun Wei Chen, Dien-Ruei Wu, Jie Lin, Yu-Kuan Fu, Yang-Hsien Lin, Eddie TC Huang, Simon See, Hung-yi Lee, Da-Shan Shiu
Abstract
Full-duplex voice interaction requires more than utterance-level conversion. It must process streaming speech, manage turn-taking and interruptions, while preserving pretrained linguistic competence and acoustic paralinguistic cues. We ask whether TASTE (Text-Aligned Speech Tokenization and Embedding) provides a viable path toward this goal. We present TASTE2, which transforms utterance-level TASTE into an incremental dialogue stack. A shared text-token vocabulary removes word-level averaging, while modality-aligned dialogue training predicts one continuous audio latent per text token without interleaving heterogeneous token streams. An incremental Speech Detokenizer enables streaming synthesis through CosyVoice2. After speech and dialogue training, TASTE2 (Merge) reaches 56.3% on LLaMA-Questions against a 57.3% Qwen2.5-7B Instruct text-only reference (98.2% accuracy retention), and TASTE2 (Direct) reaches 53.0% (92.4% retention). We build TASTE2 VoiceBot, which processes user speech incrementally, streams synthesized audio, and stops generation on barge-in. On Full-Duplex-Bench v1.0, TASTE2 and TASTE2 VoiceBot handle interruptions well while maintaining high conversational coherence. Natural conversation remains challenging, and deployed mean time to first audio is 2.701 s on two NVIDIA RTX A6000 after TensorRT acceleration. Finally, to our knowledge, we provide the first systematic characterization of explicit paralinguistic control in a TASTE based model. Fast speaking rate serves as a cross-strategy proof of concept after dialogue SFT, while emotion control is strategy dependent and the remaining attributes stay weak. Together, these results establish TASTE based modeling as a practical route toward full-duplex systems while identifying natural conversation robustness, speech generation latency, and feature general paralinguistic control as open challenges. Explore TASTE2 online.