Audio language models represent speech and text features differently
Do Audio Language Models Hear and Read Distinctive Features Alike?
Computation and LanguageMachine LearningSound
Summary
This work looks at whether audio language models interpret speech sounds and written text the same way, especially for small speech differences called distinctive features. The authors measure if the model's inner representations align similarly for heard and read versions of these features, like voicing in consonants. They find that mostly the model family influences this alignment, with only some models showing consistent similarity for certain features like voicing. This suggests that audio language models may treat speech and text inputs differently.
What this means in practice
- •For speech technology developers: Improve speech recognition systems by understanding how models align spoken and written phoneme features differently.
- •For natural language processing teams: Design better multimodal language models that process speech and text more consistently by leveraging feature representation insights.
Authors
Yuanhao Chen, Peter Chin
Abstract
Audio language models pass speech and text through a single decoder. We ask whether that decoder represents a distinctive feature in the same direction when a phoneme is heard and when it is read. For minimal pairs of phonemes differing in one feature, we take the offset between the two members' mean representations. Averaging those offsets gives a direction for each stream, and we measure the cosine between the two. Because the two streams already agree about arbitrary phoneme pairs, we compare every measure against a reference built from random pairings rather than against zero. We apply this to 6 models, 7 features and 15 languages from 11 families. Only voicing in the two Qwen2.5-Omni models exceeds that reference after correction for multiple testing, and the reference varies by a factor of seven between models. In three of the six models, voicing has one direction in audio across the 14 languages with enough minimal pairs to measure it, and every language pair agrees in two of them. The model family, not the model size, predicts which stream represents a feature.