Spoken agents learn when to act while still listening
Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems
Computation and Language
Summary
Spoken dialogue systems often wait until a person finishes speaking before they respond, which can cause delays. The authors created a large dataset that marks the earliest moment a spoken request can be understood and acted upon. They trained a model that can decide when enough has been heard to trigger a tool call earlier than waiting for the full request. This leads to faster responses, cutting action delays by over a second on average in several test situations.
What this means in practice
- •For voice assistant developers: Build voice assistants that initiate actions before users finish speaking by predicting the earliest point to call supporting tools.$Commercial implications: Enables faster, more responsive voice assistant products that improve user experience by reducing wait times for in-dialogue actions.
- •For call center software engineers: Integrate early action recognition to speed up automated responses in customer service call systems, reducing interaction latency.
Authors
Yizhou Peng, Chen Chen, Donghang Wu, Hexin Liu, Eng Siong Chng
Abstract
Latency-sensitive spoken agents must know not only which tool to call, but also when they have heard enough to act. Waiting for the turn to complete delays execution even when the request is already actionable. However, neither existing spoken agents nor spoken dialogue corpora define the earliest point at which a request becomes actionable. We introduce a synthetic spoken-dialogue pipeline that provides earliest-actionable-point supervision, marking when the observed speech grounds a tool and its required arguments. Through controlled task generation, speech synthesis and alignment, and gated annotation, we construct 26,694 dialogues totaling 786 hours across 48 domains, with disjoint training and test argument values. We additionally create an audio rendering of $τ^2$-bench for external evaluation. We train an omni-modal language model on this corpus, enabling causal, chunk-level decisions and partial-transcript generation for downstream call completion. The model achieves tool-state macro-F1 scores of 0.48 and 0.47 on held-out dialogues and the external benchmark, respectively. For correct calls, the estimated mean lead over a turn-end cascade, using the same formatter, is 1.8-2.0 seconds in-domain and 1.1 seconds on $τ^2$-bench. These results demonstrate a data-driven path toward spoken agents that act while listening. We will open-source the pipeline and the training scripts.