Automatic evaluation method measures human interpreter quality accurately
Rubric-Aligned Disentangled Evaluation of Human Simultaneous Interpreting
Computation and Language
Summary
Simultaneous interpreting is when people translate speech live, but it's hard to automatically measure how well they do it. The authors created a dataset with human scores on meaning, delivery, and timing for many short interpreted speech segments. They found that current methods mix up these evaluation parts and don't match human judgments well. They developed a new machine learning model that better keeps these evaluation aspects separate and aligns more closely with human ratings, making it useful for giving interpreters helpful feedback.
What this means in practice
- •For translation service providers: Use automated segment-level evaluation to give interpreters quick feedback on meaning and delivery quality during live sessions.
- •For ai evaluation tool developers: Incorporate disentangled dual-head supervised models for refined evaluation metrics that separate meaning accuracy from delivery and timing.
- •For language pedagogy software makers: Integrate stable ranked evaluation signals to support learners practicing live interpreting with formative feedback.$Commercial implications: Enables development of language learning apps that provide detailed, rubric-aligned scoring for simultaneous interpreting practice.
Authors
Ziyu Zhang, Satoshi Nakamura
Abstract
Human simultaneous interpreting (SI) is commonly assessed with analytic rubrics separating meaning transfer, delivery quality, and temporal synchrony, yet no automatic metric is designed for rubric-aligned segment-level SI evaluation. We construct a professionally annotated corpus of 1,101 SI segments with scores for meaning transfer (LQ), delivery quality (EXP), and perceived latency (LAT). We show that structured LLM prompting and scalar supervision collapse rubric dimensions, yielding near-zero correlation with human ratings and strong cross-dimension coupling. To isolate supervision structure under identical backbone capacity, we introduce dual regression heads on a LoRA-adapted COMET-KIWI encoder. On a held-out talk-level test set, the model achieves Pearson correlations of 0.388 (LQ) and 0.301 (EXP), improving over frozen COMET-KIWI. Given low absolute rater agreement, we interpret results relative to human consistency and target stable ranking signals for formative assessment.