Personalized speech recognition improves understanding of impaired Czech speaker
Personalized Automatic Speech Recognition for a Dysarthric and Tracheostomic Speaker using Artificial Conversations
Artificial IntelligenceComputation and LanguageHuman-Computer InteractionSound
Summary
Some people have trouble speaking clearly due to health conditions, making it hard for others to understand them. This work created a special speech recognition system tailored to a person with severe speech difficulties after a surgery, using recordings of their own voice. The system learned from both common Czech speech and simulated versions of the person's speech to do much better than standard tools. It helps recognize what the person says even when their speech is very hard to understand.
What this means in practice
- •For assistive technology developers: Create better speech recognition tools personalized to users with severe speech impairments for improved communication aids.$Commercial implications: Enables development of customized ASR-based assistive communication devices tailored to individuals with unique speech challenges.
- •For speech therapy software teams: Incorporate personalized ASR models using simulated and real impaired speech data to assess and support speech rehabilitation progress.
Authors
David Nadrchal, Monorama Swain, Florian Schmid, Gerhard Widmer, Paul Primus
Abstract
This work presents an automatic speech recognition (ASR) system personalized for a Czech speaker with a permanent tracheal stoma and severe dysarthria rendering their speech unintelligible to untrained listeners. We release a public dataset containing 33 annotated hours of the speaker's speech, collected using a novel "artificial conversation" protocol designed for high engagement and dialogue realism. We propose a multi-stage training pipeline based on Whisper Base: fine-tuning on standard Czech speech, acoustically simulated tracheostomic speech, and the speaker's data. We evaluate the system across three near real-time scenarios: scripted conversations, question answering, and spontaneous dialogue, achieving a 50\% relative reduction in Character Error Rate compared to Whisper Base baseline and surpassing the average recognition accuracy of their assistants in acoustic recognition of isolated utterances. We demonstrate that even for severely impeded speech, a helpful ASR is achievable, as evidenced by the quantitative results and the feedback from the speaker.