Language models gain audio skills without retraining their brain
Symbiotic Architecture for Post-Hoc Audio Extension of Frozen Language Models
SoundComputation and Language
Summary
Large language models can understand text very well but need extra training to handle sounds like speech or environmental noise. This paper shows how an added module can feed audio information into the model's short-term memory without changing the main model itself. This keeps the original text abilities intact while enabling new audio tasks like speech recognition and sound classification. The authors demonstrate that their approach performs nearly as well as retraining the whole model but is more efficient and avoids messing up the original skills.
What this means in practice
- •For voice assistant developers: Add audio recognition to existing text language models without expensive retraining, enabling faster deployment of voice-enabled features.$Commercial implications: Enables companies to sell upgraded voice assistants using their current language models with minimal additional training costs.
- •For audio software engineers: Create systems that integrate audio inputs into frozen language models to perform multimodal tasks while preserving original text functions.
Authors
Yotaro Kubo, Qi Sun, Yujin Tang
Abstract
This paper proposes an architecture for equipping large language models (LLMs) with audio-understanding capabilities without fine-tuning their weights. The proposed symbiotic architecture employs an injector module that writes audio-conditioned vectors directly into the target LLM's short-term memory, i.e., the key-value (KV) cache, enabling the LLM to behave as an audio language model (ALM). The architectural advantages are twofold. First, it improves the scalability of ALMs: because the proposed method bypasses the LLM during audio injection, the injection cost is governed by the injector width rather than the backbone width, and can therefore scale more slowly than the cost of full-backbone prefilling. Second, since the training scheme does not update the LLM weights, the original capabilities of the LLM are preserved without the risk of degradation from fine-tuning. The effectiveness of the proposed method is evaluated on both audio-understanding tasks (automatic speech recognition, audio question answering, and acoustic scene classification) and text-only tasks. We confirm that, while activating fewer parameters during audio prefilling, our architecture outperforms the conventional method with a frozen LLM and approaches the performance of a fine-tuned ALM, all while preserving the backbone LLM's original text-only task performance by construction.