Audio language model reveals hidden signals linked to dog barking sounds
When Does the Concept of "Dog" Emerge in an Audio LLM?
Artificial Intelligence
Summary
Figuring out how AI models understand sounds like dog barks is challenging. The authors studied a large audio language model and found a specific hidden signal that relates to the concept of a dog barking. By adding or removing this signal, they could change how the model responded to questions about dog sounds. This signal became most important just before the model gave its final answer. The work helps show how audio models represent and use sound meanings internally.
What this means in practice
- •For audio ai developers: Improve sound recognition modules by identifying internal signals corresponding to specific animal sounds in large audio language models.
- •For wildlife monitoring teams: Use insights about dog sound representations inside AI to refine automated detection of barking in environmental audio streams.
Authors
Zhe Wang, Shiqi Liu, Ruiyun Zhong, Tiechong Zhu, Yihua Tan
Abstract
Multimodal large language models answer audio questions, but how they represent auditory semantics and use them in decisions remains unclear, limiting our understanding of response formation. We study dog barking in Qwen2.5-Omni-7B using Jacobian lens (J-lens) readout and directional interventions. We define the dog direction as a J-lens-derived hidden-state vector associated with dog; adding or removing its component modulates dog-related information. We find this information decodable without dog/bark prompt cues or animal-identification requirements. Directional interventions change response tendencies and some final answers, with effects concentrated in late-layer states immediately before generation across species classification, vocalization classification, and sound description. The dog direction shows no comparable advantage over controls in animal/other classification. These results provide causal-intervention evidence that the dog direction affects output scores in a task-dependent manner, most consistently at L22 and L24 immediately before generation.