Large audio models improve task flexibility with adapter generation
In-Context Adapter Generation for Compositional Generalization in Large Audio Language Models
Artificial IntelligenceSound
Summary
Large audio language models are good at understanding speech but struggle when faced with new, combined tasks. The authors found a way to help these models handle new tasks better by creating small, task-specific adapter modules that change based on the instructions given. They built a special system called T2L-ICAG that generates these adapters in context for each task. Tests showed this method works better than earlier approaches, especially for tasks that are very different from what the model has seen before.
What this means in practice
- •For speech technology teams: Create adaptable speech recognition or understanding models that can better handle new combinations of audio tasks.
- •For audio ai developers: Develop audio processing tools that adjust to new user instructions by generating task-specific model components on demand.
Authors
Yiwei Guo, Siqi Huang, Bohan Li, Jing Peng, Hankun Wang, Zhihan Li, Kai Yu
Abstract
Large audio language models (LALMs) have achieved strong performance on standard speech and audio understanding tasks, yet their ability to systematically generalize to novel task compositions remains underexplored. Taking a meta-learning view, we extend Text-to-LoRA (T2L) to LALMs, generating instruction-conditioned LoRA adapters with a Transformer hypernetwork rather than learning a single fixed adapter shared across instructions. To exploit fine-grained task information, we introduce in-context adapter generation (ICAG), a token-level hypernetwork architecture; we call the resulting method T2L-ICAG. We evaluate this approach on two representative LALMs using SIFT-50M for supervised fine-tuning and construct a controlled compositional evaluation suite covering five task classes. Across two LALM backbones, T2L-ICAG consistently improves over static LoRA and also the global embedding-based T2L baseline. Coverage and capacity analyses further show that these gains are larger on structurally out-of-domain tasks and cannot be explained by parameter count alone. These results suggest that generating task-specific adapters from instructions is a promising direction for improving compositional generalization of LALMs.