EviSI improves evaluation of real time speech translation quality
EviSI: An Evaluation Agent for Simultaneous Interpreting
Computation and Language
Summary
Real-time speech translation has to understand, translate, and speak while someone is still talking. Usual ways to check if the translation is good may get confused because spoken translation can change words to keep up without losing meaning. The paper’s authors created EviSI, a tool that uses a language model to better judge if these translations keep the meaning and sound right. EviSI matches human opinions about translation quality better than some current tools, especially for English to Chinese.
What this means in practice
- •For machine translation developers: Improve automatic quality checks in live speech translation systems to better reflect changes that preserve meaning despite reformulation.
- •For speech technology teams: Enhance evaluation of speech-to-speech systems delivering Chinese-English translations to improve realtime service quality assessments.
Authors
Ben Yan, Zongyao Li, Daimeng Wei, Weidong Liu, Huan Zhao, Chong Li, Yaode Wang, Yuzhe Shang
Abstract
Simultaneous speech-to-speech translation requires understanding, translation and spoken delivery while the source stream continues. To support timely delivery and limit accumulated delay, systems adopt reformulation and summarization, which can preserve meaning while departing from written references. BLEU and COMET may not reliably distinguish such variation from semantic loss. We introduce EviSI, a large language model evaluation agent adapting the error analysis and penalty principles of Multidimensional Quality Metrics (MQM). It constructs shared source evidence, assesses semantic fidelity and oral expression, reconciles overlapping errors and scores deterministically. EviSI recovers the aggregate human system ranking for English to Chinese. Mean Kendall agreement with human system rankings within corpora reaches 0.707 for English to Chinese and 0.467 for Chinese to English, exceeding evaluated baselines. An extension across five directions shows positive concordance with COMET without human ratings. Individual output agreement with humans remains mixed.