Simultaneous speech translation improved by self-adapting offline models
SimPOTrans: Self-Adaptation for Offline-to-Simultaneous Speech Translation via Reinforcement Learning
Computation and Language
Summary
Translating speech as it is spoken requires balancing accuracy and speed. The authors present SimPOTrans, a method that helps offline speech translators adapt themselves to translate in real time by learning from partial speech input. This approach does not need complicated training signals or extra rules for timing. It shows better results than previous methods in translating English speech into German, Chinese, and Japanese while keeping translations quick and good quality.
What this means in practice
- •For speech technology developers: Improve real-time translation systems by adapting offline models to simultaneous scenarios without extra training data or complex policies.
- •For live event interpreters: Enable simultaneous translation software to better manage trade-offs between translation speed and accuracy by tuning input processing chunk sizes.
Authors
Haotian Tan, Hiroki Ouchi, Sakriani Sakti
Abstract
Simultaneous speech translation (SimulST) aims to translate ongoing speech with low latency while preserving translation quality. Recent studies have adopted large language models (LLMs) for SimulST and formulated it as a multi-turn dialogue task for efficient generation, achieving state-of-the-art performance. However, existing dialogue-style SimulST approaches typically rely on specialized training schemes to acquire simultaneous translation behavior, either using synthesized read/write trajectories, which may provide suboptimal training signals, or jointly optimizing translation quality and latency, whose competing objectives may cause optimization instability. In this paper, we introduce SimPOTrans, a reinforcement learning (RL) framework that enables offline speech translation models to self-adapt to dialogue-style SimulST. Through a simple two-stage RL post-training pipeline, SimPOTrans rewards an offline model for exhibiting translation behaviors beneficial to simultaneous scenarios when given partial speech inputs, thereby enabling offline-to-simultaneous self-adaptation without synthesized read/write trajectories or explicit latency optimization. At inference time, SimPOTrans requires no external decision policy and enables flexible quality--latency control by simply adjusting the chunk size of speech input. Experiments on the ACL 60/60 and MCIF benchmarks across three language directions (En$\rightarrow$De, En$\rightarrow$Zh, and En$\rightarrow$Ja) demonstrate that SimPOTrans outperforms several strong baselines, achieving a superior quality--latency trade-off.