Model balances speed and quality in streaming speech translation
S3-S2ST: Semantically Supervised Streaming Speech-to-Speech Translation
Sound
Summary
Translating spoken language in real-time is tricky because the system needs to decide when it has heard enough to give a good translation without waiting too long. The authors created a model called S3-S2ST that learns when to start translating and speaking based on the meaning of what’s been said so far. It works by turning whole speech sentences into smaller chunks for training, helping the system decide the best moment to translate. This approach allows adjusting the balance between translation accuracy and how quickly it responds. The model performs well on English-Chinese translation tasks and shows improvements in translation quality with smart adjustments.
What this means in practice
- •For voice assistant developers: Enable real-time multi-language voice assistants that decide when to translate spoken input for low delay and high accuracy.
- •For conference interpreting platforms: Improve live translation quality by dynamically adjusting when partial speech input triggers translations to better balance speed and accuracy.
- •For call center technology teams: Integrate smarter streaming speech translation policies to enhance multilingual customer service with faster and better spoken translations.$Commercial implications: Supports development of commercial real-time multilingual support products by improving translation timing and quality.
Authors
Yuxiang Zhao, Zhiyun Fan, Yu Pu, Yanjie An, Yushen Chen, Wenrui Liang, Baiji Liu, Yunchong Xiao, Zhaoci Liu, Keqi Deng, Kai Yu, Zhifu Gao, Xiangang Li, Xie Chen
Abstract
Streaming speech-to-speech translation requires timely decisions about when partial input is sufficient for accurate translation. We present S3-S2ST, a model that learns to translate and speak from semantically aligned streaming supervision. By converting utterance-level speech pairs into incremental training examples, we teach a learned policy when to commit translations. Its adjustable threshold controls the quality--latency tradeoff on a fixed 640\,ms input grid. S3-S2ST delivers competitive English--Chinese translation on FLEURS and leads the evaluated streaming systems in translation quality on the internal 9-pair test set. On RealSI, adapting the threshold to buffered speech further improves quality over a low fixed threshold with little change in first-audio latency. We plan to release the model and code.