Evaluation agent improves scoring of real-time speech translation quality
EviSI: An Evaluation Agent for Simultaneous Interpreting
Computation and Language
Summary
Real-time speech translation is hard because the system must listen, translate, and speak all at once without much delay. Traditional ways to check translation quality don’t always work well when the translation changes the original wording to keep up with speaking speed. The authors created EviSI, an AI tool that looks at the meaning and how well something is spoken, matching human judgments better than other methods. EviSI helps to judge translations between English and Chinese more fairly by considering how the meaning is kept, even if the words differ.
simultaneous translationspeech-to-speech translationsemantic fidelityMultidimensional Quality Metrics (MQM)BLEU scoreCOMET metriclarge language modelevaluation agentKendall rank correlationEnglish-Chinese translation
Authors
Ben Yan, Zongyao Li, Daimeng Wei, Weidong Liu, Huan Zhao, Chong Li, Yaode Wang, Yuzhe Shang
Abstract
Simultaneous speech-to-speech translation requires understanding, translation and spoken delivery while the source stream continues. To support timely delivery and limit accumulated delay, systems adopt reformulation and summarization, which can preserve meaning while departing from written references. BLEU and COMET may not reliably distinguish such variation from semantic loss. We introduce EviSI, a large language model evaluation agent adapting the error analysis and penalty principles of Multidimensional Quality Metrics (MQM). It constructs shared source evidence, assesses semantic fidelity and oral expression, reconciles overlapping errors and scores deterministically. EviSI recovers the aggregate human system ranking for English to Chinese. Mean Kendall agreement with human system rankings within corpora reaches 0.707 for English to Chinese and 0.467 for Chinese to English, exceeding evaluated baselines. An extension across five directions shows positive concordance with COMET without human ratings. Individual output agreement with humans remains mixed.