Papers for

language pedagogy software makers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Automatic evaluation method measures human interpreter quality accurately

Rubric-Aligned Disentangled Evaluation of Human Simultaneous Interpreting

Abstract: Human simultaneous interpreting (SI) is commonly assessed with analytic rubrics separating meaning transfer, delivery quality, and temporal synchrony, yet no automatic metric is designed for rubric-aligned segment-level SI evaluation. We construct a professionally annotated corpus of 1,101 SI segments with scores for meaning transfer (LQ), delivery quality (EXP), and perceived latency (LAT). We show that structured LLM prompting and scalar supervision collapse rubric dimensions, yielding near-zero correlation with human ratings and strong cross-dimension coupling. To isolate supervision structure under identical backbone capacity, we introduce dual regression heads on a LoRA-adapted COMET-KIWI encoder. On a held-out talk-level test set, the model achieves Pearson correlations of 0.388 (LQ) and 0.301 (EXP), improving over frozen COMET-KIWI. Given low absolute rater agreement, we interpret results relative to human consistency and target stable ranking signals for formative assessment.

Thu 10 SeptComputation and Language

The gist

Simultaneous interpreting is when people translate speech live, but it's hard to automatically measure how well they do it. The authors created a dataset with human scores on meaning, delivery, and timing for many short interpreted speech segments. They found that current methods mix up these evaluation parts and don't match human judgments well. They developed a new machine learning model that better keeps these evaluation aspects separate and aligns more closely with human ratings, making it useful for giving interpreters helpful feedback.

Open → 2609.11131v1