SkillFormer improves audio models by splitting and routing distinct skills
SkillFormer: Skill-Decomposed Adaptation for Audio Language Models
SoundMachine Learning
Summary
Audio language models need to perform many different tasks, like recognizing emotions or counting speakers, but training them all together can cause problems where improving one task hurts another. The authors propose SkillFormer, which breaks down audio understanding into separate skill-specific components that are combined dynamically based on the question being asked, so each task uses its own tailored parameters. This design prevents conflicts during training and improves overall accuracy without major changes to existing models. They tested SkillFormer on different models and datasets, showing consistent improvements across a range of audio understanding abilities.
What this means in practice
- •For speech technology teams: Enhance multi-skill audio recognition systems by dynamically activating specialized components tailored to each audio query type, improving accuracy and efficiency.
- •For music analysis engineers: Improve automated musical feature extraction like pitch and tempo estimation by selectively applying skill-specific model adapters.
- •For customer service platform developers: Develop smarter sentiment and emotion recognition services that switch between specialized audio analysis skills for better understanding of caller mood.$Commercial implications: This enables development of advanced emotion-aware customer support tools by integrating skill-specific audio analysis, differentiating the product in the market.
Authors
Lee Seung-woo, Bowen Qi, Kim Min-jun, Jang Won-young
Abstract
Audio language models must handle dozens of distinct skills, from pitch comparison and speaker counting to musical tempo estimation and emotion recognition. Joint training on all skills at once causes interference: gains on one skill often come at the cost of another. We propose \textbf{SkillFormer}, which decomposes audio understanding into skill-specific low-rank adapters and composes them at inference time through a learned router. The router examines the question to decide which adapters to activate and how much weight each should carry, so that a pitch query engages different parameters than a genre classification query. An alternating training schedule updates each adapter on its own skill cluster before jointly calibrating the router, preventing the gradient conflicts that arise in standard multi-task optimization. SkillFormer adds fewer than 4\% of the base model's parameters and requires no changes to the audio encoder or language backbone. Evaluated on three architecturally distinct models across MMSU, MMAU-Pro, and MMAR, it raises the average accuracy by 2.5 to 4.1 points, with balanced gains across perception, reasoning, and semantic subcategories.