Situational distortion metric improves speech recognition error scoring
SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation
Computation and Language
Summary
Typical speech recognition errors are judged mostly by matching words, but this doesn’t always capture how serious the errors are in changing the meaning or intent of a sentence. The authors propose SITAR, a new scoring method that measures how much errors impact the situation described by the speech. They build SITAR using a special formula and train a light BERT-based model that scores errors quickly without needing big language models during use. Their tests show SITAR reliably tracks speech recognition quality and gives useful extra insight about errors, which could help improve automatic transcription systems.
What this means in practice
- •For speech recognition engineers: Evaluate ASR outputs with a lightweight metric that reflects the true impact of errors on meaning and intent in spoken dialogue.
- •For voice assistant developers: Improve dialogue system debugging by diagnosing which recognition errors affect conversational outcomes most significantly.
Authors
Mingzhe Lu, Jinhao Liu, Yunpeng Li, Hang Su, Lichun Fan, Jian Luan
Abstract
In automatic speech recognition (ASR), transcription errors with similar lexical or semantic discrepancies can have markedly different consequences for the situation conveyed by speech. Capturing these differences requires not merely contextual assessment, but an understanding of what the spoken discourse describes, establishes, or seeks to bring about. However, relying on LLM judges introduces computational cost and scoring variability. We introduce SITAR, a situational distortion metric for ASR, realized through a comprehensive end-to-end construction framework. Specifically, this framework aggregates situational severity and cumulative extent into a unified score via a bounded recursive formulation. Using targets derived from this formulation with limited offline LLM support, we train SITAR as a lightweight BERT-family regressor that scores reference-hypothesis pairs in a single forward pass, entirely free of any prohibitive runtime costs. Recognizing that introducing a new metric demands rigorous justification, we formalize a principled validation protocol spanning both extrinsic validity, assessed at macro and micro levels, and intrinsic scorer reliability. The former, evaluated across diverse datasets and models, demonstrates that SITAR reliably tracks broad capability scaling while providing orthogonal diagnostic utility, whereas the latter, examined on a purpose-built controlled benchmark, confirms the operational soundness of the scoring instrument itself. Through systematic validation, we demonstrate that SITAR holds strong potential as a practical, lightweight complement to existing ASR metrics. We will release the full recipes (code, model weights, dataset, etc.) after the internal review to support community research.