Large audio language models align sound and text through shared neurons
Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons
SoundComputation and Language
Summary
Some large models that work with both sound and language can understand things like emotion or speaker gender from audio, even though they learn language from text alone. The authors looked for neurons inside these models that respond to both sound and text, showing that the two types of information are connected in a shared space. By experimenting with these neurons, they found they could influence how the model interprets the audio and text signals. This shows how audio and text are intertwined inside big language models that also handle sound.
What this means in practice
- •For speech technology developers: Improve emotion and speaker attribute recognition by targeting neurons that link audio and text representations in models.
- •For assistive technology designers: Design interfaces that adapt audio understanding features using manipulable neurons blending text and sound insight.
Authors
Tzu-Quan Lin, Chih-Kai Yang, Yi-Cheng Lin, Hao Tang, Hung-yi Lee
Abstract
While the language model backbone is pretrained on text alone, large audio-language models (LALMs) can, to a great extent, recognize acoustic attributes, such as emotion, speaker gender, and spoken language. However, it is unclear how the audio representation is integrated, if at all, with the other text representation before the model produces text. We hypothesize that there is a shared subspace where the audio and text representations are aligned. To test this hypothesis, we identify cross-modal neurons that are responsible for representing both audio and text. We find alignment between audio and text neurons to various degrees across three LALMs and four acoustic attribute types. Causal interventions, steering and deactivating the neurons, confirm that the cross-modal neurons are not just necessary for prediction but can also be manipulated.