How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation
2026-03-19 • Computation and Language
Computation and LanguageSound
AI summaryⓘ
The authors investigated how much hearing-related knowledge large language models (LLMs) learn just from text, and how this affects their ability to work with audio. They tested different LLMs by checking their knowledge using a special audio-related benchmark, by seeing how well they understand text descriptions of sounds, and by training them directly with audio data. They found that some LLMs know more about sound than others, and that knowing more from text alone helps when working with audio later. Their study helps explain how LLMs can be better used in audio research.
Large Language ModelsAuditory KnowledgeAudio-Grounded EvaluationAudio CaptioningBenchmarkingFine-TuningAudio EncoderCascade EvaluationText-Only Pre-training
Authors
Ke-Han Lu, Szu-Wei Fu, Chao-Han Huck Yang, Zhehuai Chen, Sung-Feng Huang, Chih-Kai Yang, Yi-Cheng Lin, Chi-Yuan Hsiao, Wenze Ren, En-Pei Hu, Yu-Han Huang, An-Yu Cheng, Cheng-Han Chiang, Yu Tsao, Yu-Chiang Frank Wang, Hung-yi Lee
Abstract
Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.