Fa-bench compares word and phone alignment in clean and noisy speech
FA-Bench: A Benchmark for Word-Level and Phone-Level Forced-Alignment and ASR Timestamps Under Clean and Noisy Conditions
Computation and LanguagePerformance
Summary
Forced alignment finds the timing of words and sounds in a spoken audio given a transcript, but different studies have used different methods that make it hard to compare results. The authors created FA-Bench, a shared tool and set of rules to test many alignment systems fairly on the same audio, both clean and noisy. They tested 21 open models and 9 commercial speech recognition services, showing how well each matches the transcript timing and where errors happen. They also found some systems consistently mistime words by a small but consistent delay or advance. FA-Bench's code and updated results are freely available.
What this means in practice
- •For speech technology engineers: Benchmark forced alignment systems under consistent conditions to choose best timing for audio transcripts.
- •For voice assistant developers: Improve timing accuracy of word recognition in noisy environments using unified evaluation of ASR timestamps.
Authors
Wei Chu, Yuanzhe Dong, Ke Tan, Dong Han, Yichao Zhou, Ruchao Fan, Bingshen Mu, Jingbei Li, Vishwas Shetty, Sarthak Bisht, Ziyue Qiu, Massa Baali, Rita Singh, Bhisha Raj
Abstract
Forced alignment aligns speech audio with a text transcript to generate word and phone timestamps. Published comparisons normalize transcripts, split the data and match boundaries differently, so their numbers cannot be read together. We present FA-Bench, an open framework that fixes those choices once and releases the code, splits, phone mapping, text normalization and scoring script, with results published periodically. Track 1 gives every aligner the reference transcript and Track 2 gives it a recognizer's output, on the same audio, clean and degraded four ways, with 21 open models and 9 commercial APIs under a unified protocol. We score every boundary of an utterance and check the two labels beside it, so a word the recognizer missed or invented is charged. Using a tolerance-based F1 as our primary metric eliminates the 9% to 14% score inflation that standard MAE causes on recognition-dependent systems in conversational speech. We then group boundaries by their position and how many adjacent words were recognized correctly, which shows where a system lost the score. We discovered systematic bias in how current systems time words, with Whisper about 150 ms early and several commercial ASR APIs over 50 ms late. Code and results are at https://github.com/olewave/fa-bench