Speech representation learning improves with graph knowledge transfer
Enhancing speech representation learning with cross-modal knowledge transfer with HGNN under low resource settings: the case study of Yemba
Computation and Language
Summary
Learning useful speech features is hard for languages with little data. The authors propose a method that connects sounds and language parts using a special graph, helping speech features become clearer and more meaningful. Their approach allows language information to help improve sound understanding, especially when data is scarce. They tested it on English and a Cameroonian language and found improvements in recognizing isolated words.
What this means in practice
- •For speech ai developers: Improve speech recognition accuracy in languages with limited audio training data by integrating linguistic information with acoustic features using graph-based models.
- •For language technology teams: Build better speech processing tools for underrepresented languages by transferring language knowledge to audio features through heterogeneous graph neural networks.
Authors
Yannick Yomie Nzeuhang, Paulin Melatagia Yonta, Marie Tahon
Abstract
Acoustic representation learning is crucial for speech processing, yet low-resource languages (LRLs) face severe data scarcity, limiting the effectiveness of traditional and self-supervised methods. As a promising alternative, in this work, we propose to enhance acoustic representation trough a cross-modal transfer knowledge approach, based on heterogeneous graph neural networks (HGNNs), where acoustic and linguistic entities are modeled as distinct node types within a unified graph. Through message-passing mechanisms, linguistic nodes explicitly transfer knowledge to acoustic nodes, enabling structured and interpretable cross-modal information flow. To highlight this knowledge transfer and its benefits, we measured standard clustering metrics as an intrinsic evaluation of acoustic representation, and to emphasize applicability, we performed isolated-word recognition tasks using an English benchmark and a Cameroonian language dataset in low resources settings . Results demonstrate that acoustic representations consistently benefit from linguistic knowledge propagated through the graph. To our knowledge, this is the first demonstration of explicit cross-modal knowledge transfer for acoustic representation learning using HGNNs, highlighting a promising direction for speech representation in low-resource settings.