Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models
2026-08-03 • Multimedia
MultimediaArtificial Intelligence
AI summaryⓘ
The authors study how audio-based AI can better solve complicated tasks by using tools and reasoning instead of just listening. They created SpeechAgent-R, a system that learns how to interact with tools and skills through lots of training data and improves with trial and error. They also built a big dataset and a testing benchmark to see how well such agents perform on many tasks, including new ones they haven't seen before. Their results show that teaching the agent to coordinate tools and skills helps it work better in varied situations.
audio reasoningmultimodal understandingreinforcement learningsupervised fine-tuninginteraction trajectoriestool usageworkflow compositionout-of-distribution generalizationbenchmark dataset
Authors
Yuwen Wang, Tian-Hao Zhang, Minghao Cai, Yilin Ren, Ziyang Jiang, Xin Wang, Zhichao Wang, Pan Zhou, Kun Zhan, Xinyuan Qian
Abstract
Complex acoustic problems may require models to perform acoustic operations, interact with external tools and reason over the resulting textual or processed-audio observations rather than answer directly from a fixed audio input. We study such problems as tool-interactive audio reasoning and develop SpeechAgent-R, an audio agent that coordinates its intrinsic multimodal understanding with external skills and tools. To support this capability, we construct HIU-Corpus, comprising 65,492 interaction trajectories and 507.6 hours of audio across 24 tasks, 8 skills and 9 tools. SpeechAgent-R first learns structured interaction behaviors through trajectory-based supervised fine-tuning and then improves its decisions through multi-turn reinforcement learning. We further introduce HIU-Bench to jointly evaluate task performance, interaction quality and generalization to diverse task settings. It contains 1,395 samples across 56 tasks, including in-distribution (ID) and out-of-distribution (OOD) splits with substantial shifts in tool usage and workflow composition. SpeechAgent-R achieves 84.17 on ID tasks and 70.94 on OOD tasks, improving over the base model under the same agent harness by 15.40 and 14.23 points. These results demonstrate that learning skill and tool coordination improves audio agents' ability to handle diverse task settings and adaptive tool interactions.