Spoken Function Calling: A New Perspective on Spoken Language Understanding for Large Audio Language Models
2026-08-05 • Computation and Language
Computation and LanguageMultimedia
AI summaryⓘ
The authors address a problem in spoken language understanding (SLU), which helps computers understand spoken commands. Traditional SLU works well for specific tasks but struggles with more open or flexible situations. They introduce a new method called Spoken Function Calling (SFC), which uses clear rules to better interpret spoken inputs. By creating new datasets and testing advanced language models, they show that SFC improves how accurately these models understand spoken language compared to older methods.
Spoken Language UnderstandingTask-oriented Dialogue SystemsIn-context LearningLarge Language ModelsLarge Audio Language ModelsSemantic ExtractionSpoken Function CallingMulti-agent SystemsSFC-Bench Dataset
Authors
Yuezhang Peng, Yuxin Liu, Changfeng Gao, Zhifu Gao, Xiangang Li, Xie Chen
Abstract
Spoken Language Understanding (SLU) is the core component of task-oriented dialogue systems and a pivotal link in achieving seamless human-agent interaction. While traditional SLU can effectively extract user semantics for closed-set tasks after in-domain supervised fine-tuning, it faces significant challenges in leveraging in-context learning for open-domain tasks due to its ambiguous rule definitions. This work proposes Spoken Function Calling (SFC), a novel semantic understanding perspective that optimizes semantic understanding with structured rule definitions, to evolve beyond traditional closed-set SLU. Specifically, we curate and extend a suite of spoken functions based on traditional SLU datasets, construct a multi-agent system to synthesize the SFC-Bench dataset, evaluate the performance of Large Language Models (LLMs) and Large Audio Language Models (LALMs), and enhance the SFC capabilities of LALMs through post-training. Experiments demonstrate that SFC outperforms traditional SLU, substantially enhancing the semantic extraction accuracy for LLMs and LALMs.