Tts system generates overlapping two-speaker dialogue with natural timing
KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interaction
SoundComputation and Language
Summary
Making computer voices that talk like people in natural conversations is hard, especially when two people speak at the same time or interrupt each other. The authors created a system called KABURI-TTS that can generate speech for two speakers separately but in a way that sounds like real overlapping talks. It uses detailed timing of sounds (phonemes) and when each speaker talks to decide how to sound. Tests showed it sounds more natural and captures the way people interrupt or talk over each other better than older methods.
What this means in practice
- •For voice assistant developers: Create more natural conversational agents that can speak with overlapping responses and interruptions like humans do.$Commercial implications: Enables sale of virtual assistants that handle realistic back-and-forth speech with timed overlaps, improving user interaction quality.
- •For game audio programmers: Generate simultaneous dialogue for two characters in games that mimics natural overlapping speech.
Authors
Ryuichiro Higashinaka, Shinnosuke Takamichi, Tetsuji Ogawa
Abstract
Realizing full-duplex spoken dialogue requires large amounts of two-channel, one-speaker-per-channel conversational speech data. Although conversational text-to-speech (TTS) engines have been developed, they are not necessarily robust to two-party simultaneous phenomena such as backchannels, interruptions, and overlaps that occur while the interlocutor is speaking. In this work, aiming at conversational speech synthesis that reproduces human-like overlap, we propose KABURI-TTS. KABURI-TTS takes a per-speaker phoneme raster as input and renders the speech of the two speakers on separate channels, conditioned on the per-frame phonemes and the voice activity derived from them. Because the phoneme raster is supplied by a separate module, the proposed method enables controllable generation of one-speaker-per-channel, two-party spoken dialogue. A user evaluation shows that, compared with strong baselines, the proposed method attains higher naturalness at both the utterance and the interaction level. Furthermore, an analysis of voice activity confirms that the proposed method produces more overlap and more frequent turn-taking.