Test-time adaptation improves sentence speech recognition for dysarthric speakers
Advancing Sentence-Level Dysarthric Audio-Visual Speech Recognition via Test-Time Adaptation
Sound
Summary
People with speech difficulties caused by conditions like dysarthria often have trouble being understood by computers because their speech varies a lot. The authors created a way to use video and audio together to recognize what these speakers say at the sentence level, which is more useful than just recognizing single words. They also developed a technique called test-time adaptation that helps the system learn from a new speaker’s speech without needing extra labeled data. This approach improved understanding of new speakers the system hadn’t seen before, making speech recognition more flexible and effective for people with speech impairments.
What this means in practice
- •For speech technology developers: Improve assistive speech recognition systems to better understand people with speech impairments without needing speaker-specific training.
- •For hearing aid manufacturers: Integrate adaptive audio-visual speech recognition methods to enhance communication aids for users with variable speech patterns.
Authors
Shiyao Wang, Hui Wang, Jiaming Zhou, Shiwan Zhao, Yong Qin
Abstract
Dysarthric audio-visual speech recognition (DAVSR) enhances performance by integrating complementary visual cues. However, existing research is largely confined to word-level tasks with limited vocabularies, which severely restricts real-world communication for dysarthric speakers. Furthermore, performance remains heavily dependent on supervised fine-tuning with speaker-specific data; significant variations in age, etiology, and severity cause traditional methods to struggle with unseen speakers. To address these domain shifts and the scarcity of continuous multimodal resources, we first introduce a "source speech search-segmentation" pipeline to construct a structured sentence-level corpus from raw CDSD videos. We then propose the use of unsupervised Test-Time Adaptation (TTA) for sentence-level DAVSR using Sequential-level Generalized Entropy Minimization (SGEM). Our evaluations across the state-of-the-art AV-SUPERB and ViSpeR frameworks demonstrate that TTA consistently alleviates Character Error Rate (CER) degradation for unseen pathological speakers. Paired t-tests verify that these improvements are statistically significant (p < 0.05), highlighting TTA as a promising paradigm for advancing speaker-adaptive assistive speech interfaces.