Forced alignment improved for noisy natural speech with laughter and fillers
Noise-Robust Forced Alignment for Spontaneous Speech
Machine Learning
Summary
Automatically matching spoken words to audio is tricky when speech is casual, with noises like laughter and filler sounds disrupting timing. The authors improve an existing tool by using modern audio features and special units to represent these natural speech sounds. Their method learns from real conversations without needing hand-corrected timings and performs better at marking word boundaries even in noisy, disfluent speech. This helps reduce big mistakes in aligning speech to text compared to standard tools.
What this means in practice
- •For speech technology developers: Improve alignment accuracy of spontaneous, noisy speech transcripts in voice assistants and transcription services.
- •For conversation analysis teams: Analyze natural conversations more precisely by accurately timing fillers and laughter in speech data processing.
Authors
Laurin Wagner, Mario Zusag
Abstract
We show that forced aligners degrade on spontaneous, noisy real-world speech, and that fillers, laughter, and disfluencies corrupt word boundaries, impacting many downstream applications. We revisit the classical GMM-HMM aligner with upsampled WavLM features instead of MFCCs, explicit phone units for fillers and non-lexical vocal events trained on verbatim-annotated spontaneous speech, and a discriminative projection fitted on clean and noise-corrupted feature pairs to improve robustness. The pipeline is self-bootstrapped from 149h of curated audio without manual or external boundary targets. On conversational (Buckeye) and disfluent (FluencyBank) speech it reaches word-boundary MAEs of 20.6 and 76.9ms, 19% and 17% lower than the strongest of ten baselines. Catastrophic misalignments (>200ms) are 2.7 times rarer than Montreal Forced Aligners (MFA) best model even after widening MFA's search beam, which we show removes a large pruning artifact from its conversational error; event units halve the error next to laughter, while being meaningfully more robust than existing aligners.