Synthesizing silent speech echoes improves wearable speech recognition accuracy
Synthesizing Ultrasonic Echo Profiles from Text for Open-Vocabulary Silent Speech Recognition
Sound
Summary
Silent speech recognition lets people communicate by moving their mouths without making sounds, using sensors to detect these movements. The authors created a method that generates fake sensor readings from text, allowing the system to learn without needing expensive real data. This synthetic data helped improve the accuracy of recognizing silent speech over many words, making wearable devices that read silent speech more usable. Their results bring the error rate down to a level where users usually feel comfortable using such devices.
What this means in practice
- •For wearable device engineers: Improve silent speech recognition systems in wearable ultrasound devices by training on synthetic echo profile data from text.
- •For assistive technology developers: Build more accurate silent speech interfaces for people unable to speak aloud by leveraging synthetic ultrasonic data for training.$Commercial implications: Enables development of market-ready silent speech devices aiding communication for speech-impaired users.
Authors
Jiacheng Liu, Ruidong Zhang, François Guimbretière, Cheng Zhang
Abstract
Eyewear that senses articulation with inaudible ultrasound can now recognize silent speech over an open vocabulary, but its accuracy is limited by the cost of training data. We present SoniGAN, which synthesizes the ultrasonic echo profiles of silent articulation from text and uses them as synthetic supervision. A voice-cloned text-to-speech (TTS) model reads a text corpus, and a two-stage generator synthesizes echo profiles from representations of the spoken content of that audio. Pretraining a recognizer on 104 hours of the synthesized data improves the word error rate (WER) of a public open-vocabulary benchmark by 2.3 points. Adding the sessions the corpus discards for a faulty microphone brings the system to 22.2% without any language model. Lexicon-constrained decoding, LLM rescoring, and ensembling further bring it to 14.9%, the first minimally obtrusive silent speech interface to fall inside the error range in which users report no loss of willingness to use silent speech input.