Adaptive dual-system voice agents improve real-time conversation accuracy
SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents
Computation and Language
Summary
People want voice assistants that can talk naturally and quickly, but also think carefully when needed. The authors created a voice agent called SALMONN-duo that uses two systems: one talks fast to keep the conversation natural, and the other thinks slowly to handle tricky questions. The fast system learns when to answer right away and when to ask the slow system for help, keeping conversations smooth and accurate. This design helps the agent do better on complex questions and tasks while staying responsive.
What this means in practice
- •For voice assistant developers: Build voice agents that balance instant replies and complex reasoning without disrupting conversation flow.
- •For customer service technology teams: Create virtual agents capable of handling multi-turn, knowledge-based dialogues in business scenarios with better accuracy and safety.$Commercial implications: Enables improved AI customer agents that comply with company policies and improve task success, making them market-ready.
Authors
Wenyi Yu, Siyin Wang, Terumi Chiba, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Chao Zhang
Abstract
Full-duplex speech large language models (LLMs) enable low-latency, natural voice interaction. However, real-world agents must also use tools and perform deliberative reasoning-operations whose variable latency and computational cost conflict with the stringent timing requirements of real-time conversation. To reconcile these demands, we propose SALMONN-duo, an adaptive dual-system voice agent inspired by dual-process theories of cognition. SALMONN-duo separates real-time interaction from deliberative computation by pairing an always-on, fast-thinking full-duplex speech LLM (system 1) with a powerful asynchronous slow-thinking LLM agent (system 2). Beyond handling real-time interaction, system 1 learns when to answer directly and when to delegate, remaining responsive during backend execution and seamlessly integrating returned information into the ongoing dialogue without exposing tool traces or losing conversational context. Evaluations on single-turn spoken question answering (QA) and multi-turn conversations demonstrate that adaptive delegation substantially improves accuracy on knowledge-intensive and multi-hop reasoning questions, while knowledge-boundary-aware training avoids unnecessary system 2 invocations. On a customized version of $τ$-Voice, SALMONN-duo further demonstrates its ability to complete environment-grounded, policy-constrained tasks through multi-turn interactions in realistic business scenarios. Finally, cost-aware reinforcement learning further enhances the trade-off between task performance and backend usage across the QA and conversation tasks, while improving task success and response safety on $τ$-Voice with an acceptable increase in the delegation rate.