Speech models encode word identity beyond phonemes in later layers
Do speech foundation models really learn words?
Computation and LanguageSound
Summary
Speech models like HuBERT and wav2vec 2.0 can recognize individual words, but this could just mean they identify sounds rather than the words themselves. The authors found that these models do actually learn representations of whole words that are separate from the sounds they contain, especially in later layers of the model. They used a method to remove the influence of phonemes and showed that the models still keep important word information. This helps improve tasks that try to find words in speech based on meaning and structure, not just sound.
What this means in practice
- •For speech recognition developers: Improve speech recognition systems by incorporating models that represent whole words separately from phonetic details.
- •For language model engineers: Enhance speech-aware language models by using disentangled word representations that capture identity beyond sound patterns.
Authors
Robin Huo, Ewan Dunbar
Abstract
Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the basis for tokens in speech-aware language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words. However, discriminative ability for words need not imply specialized representation of words per se. Good discrimination of words may be explained by good encoding of word form (phonemes) rather than form-independent word representations encoding identity or syntactic/semantic properties. By partialling out phoneme information using residualization, we show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content. We show that this simple approach to disentanglement can enhance higher-order linguistic information in word discovery tasks.