TASTE2 advances continuous real time speech for natural voice interaction
TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction
Sound
Summary
Understanding and responding to spoken conversation is tricky because people often speak over each other and change tone or emotion. The researchers developed TASTE2, a system that processes speech as it happens rather than waiting for full sentences. It also tries to keep the meaning and voice features like speed or emotion intact while allowing interruptions. Their system showed good accuracy when answering questions and handled interruptions well, though making conversations sound natural and fast still needs work. They also explored how to control voice emotions and speed, which is a first for this type of technology.
full-duplex voice interactionspeech tokenizationstreaming speech synthesisparalinguistic cuesincremental processingturn-takingspeech detokenizerdialogue trainingvoicebotTensorRT acceleration
Authors
Yi-Chang Chen, Chun Wei Chen, Dien-Ruei Wu, Jie Lin, Yu-Kuan Fu, Yang-Hsien Lin, Eddie TC Huang, Simon See, Hung-yi Lee, Da-Shan Shiu
Abstract
Full-duplex voice interaction requires more than utterance-level conversion. It must process streaming speech, manage turn-taking and interruptions, while preserving pretrained linguistic competence and acoustic paralinguistic cues. We ask whether TASTE (Text-Aligned Speech Tokenization and Embedding) provides a viable path toward this goal. We present TASTE2, which transforms utterance-level TASTE into an incremental dialogue stack. A shared text-token vocabulary removes word-level averaging, while modality-aligned dialogue training predicts one continuous audio latent per text token without interleaving heterogeneous token streams. An incremental Speech Detokenizer enables streaming synthesis through CosyVoice2. After speech and dialogue training, TASTE2 (Merge) reaches 56.3% on LLaMA-Questions against a 57.3% Qwen2.5-7B Instruct text-only reference (98.2% accuracy retention), and TASTE2 (Direct) reaches 53.0% (92.4% retention). We build TASTE2 VoiceBot, which processes user speech incrementally, streams synthesized audio, and stops generation on barge-in. On Full-Duplex-Bench v1.0, TASTE2 and TASTE2 VoiceBot handle interruptions well while maintaining high conversational coherence. Natural conversation remains challenging, and deployed mean time to first audio is 2.701 s on two NVIDIA RTX A6000 after TensorRT acceleration. Finally, to our knowledge, we provide the first systematic characterization of explicit paralinguistic control in a TASTE based model. Fast speaking rate serves as a cross-strategy proof of concept after dialogue SFT, while emotion control is strategy dependent and the remaining attributes stay weak. Together, these results establish TASTE based modeling as a practical route toward full-duplex systems while identifying natural conversation robustness, speech generation latency, and feature general paralinguistic control as open challenges. Explore TASTE2 online.