Tandem speech models learn natural conversation with randomized guidance
Learning Natural Conversational Behavior in Tandem Speech-to-Speech Models with Randomized Guidance
Computation and Language
Summary
A problem with making speech-to-speech systems is that the computer can’t see the examples of how it should respond to someone while they are still talking. The authors worked on a way to teach the model from real conversations by mixing correct responses with random ones during training. This helps the system learn to pay attention to useful hints while ignoring irrelevant ones, making it sound more natural. Their method works well on both made-up and real conversations, improving naturalness without hurting response quality.
What this means in practice
- •For voice assistant developers: Create conversational agents that take turns and respond more naturally by training with randomized response guidance from real dialogues.
- •For call center automation teams: Improve speech-driven customer service bots to handle flexible conversations and provide timely, relevant replies by using tandem models with randomized guidance.
Authors
Manato Yaguchi, Yotaro Kubo, Hikaru Asano, So Kuroki
Abstract
Tandem speech-to-speech architectures couple a responsive speech frontend with an asynchronous text backend. In KAME, a large language model (LLM) serves as the backend, supplying candidate responses as guidance to the speech frontend while the user is still speaking. Ordinary conversation recordings capture the eventual response but not the guidance the backend would supply during the user's utterance. Generating the missing guidance with a simulator LLM adds substantial data-preparation overhead when training on real conversations. We propose randomized intermediate guidance, which derives guidance directly from the conversation corpus rather than simulating backend LLM behavior. During training, target responses provide informative guidance, while randomly sampled responses provide potentially irrelevant updates during the utterance. This combination aims to teach the frontend to use backend information selectively. On synthetic dialogues, KAME trained with this recipe achieves response quality comparable to that of the LLM-generated and similarity-based baselines. Training on 3.8k hours of real conversations improves smooth turn-taking and audio-judge naturalness over synthetic-data KAME while retaining a response-quality advantage over Moshi. These results show that randomized guidance offers a practical route to combining the response-quality benefits of tandem models with natural conversational behavior learned from real speech.