Efficient speech and text alignment improves word retrieval for low-resource languages
Low resource cross-modal alignment using HGNN to enhance speech representation
Computation and Language
Summary
It can be hard to teach computers to understand speech in languages with little data available. The authors developed a method using a special type of neural network called a Heterogeneous Graph Neural Network to connect spoken words and written text more efficiently. This method works well even when there isn’t much data and helps the computer to better understand spoken words in both English and a low-resource language called Yemba. Their approach matches or beats the best existing methods while needing less data and computing power.
What this means in practice
- •For speech recognition developers: Improve word-level speech recognition models in low-resource languages using more efficient data alignment techniques.
- •For mobile app developers: Develop speech-enabled apps for underrepresented languages by using this low-data method to better map spoken and written words.
Authors
Yannick Yomie Nzeuhang, Marie Tahon, Paulin Melatagia Yonta
Abstract
Speech-text space alignment is a multimodal representation learning method consisting to map different speech and text into a shared representation space, leading to enrichment of the representation of each modality. Proposed architectures, such as SAMU-XLSR, typically follow a student/teacher framework, with the goal of fine-tuning an audio encoder to produce representations that closely match those of the text. In this way a speech representation is semantically enriched. However, such systems generally require large amounts of training data and considerable computational resource, making them difficult to apply to low resources languages under frugal constraints. The present work proposes a data-efficient space alignment method based on Heterogeneous Graph Neural Networks and link prediction. The core idea is to leverage message passing to explicitly transfer information from the text modality to the speech modality, thereby reducing the need for large training datasets and intrinsically enriching the acoustic representations, all in a more interpretable manner. Although thoroughly explored for high-resource languages, word-level tasks in speech remain relevant for certain low-resource languages. Therefore, we conducted experiments on speech-text alignment at the word level using the TIMIT (English) dataset and Yemba (a Cameroonian language). Our approach yields results comparable to those of SAMU-XLSR, a state-of-the-art method, and even surpasses it for the Yemba language in the task of word retrieval, while using far fewer resources, demonstrating its power, frugality, and efficiency.