Full duplex speech models gain real time external knowledge access

Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends

Computation and LanguageSound

Summary

Many current speech dialogue models can both listen and talk at the same time, but they often lack the ability to fetch and use up-to-date information from outside sources during a conversation. The authors introduce Context Spanning, a method that lets these speech models connect with large language models (LLMs) to bring in fresh information without losing details in the process. By feeding the external information directly and in real-time, the speech model can better understand and use it when responding. This approach improves performance on conversation benchmarks and question answering tasks, showing it helps make interactions more natural and informative.

What this means in practice

  • For voice assistant developers: Integrate real-time external knowledge into full-duplex speech assistants for more accurate and informed responses during conversations.$Commercial implications: Enables advanced voice assistants capable of simultaneous listening and speaking with up-to-date information access, meeting demand for smarter AI assistants.
  • For call center automation teams: Improve automated agents’ ability to provide timely, detailed answers by injecting external data directly into their speech models during live calls.

Authors

Seonghyeon Go, Yongwoo Kim, Hyeonjin Cha, Jaeho Shin

Abstract

Full-duplex spoken dialogue models can listen and speak simultaneously like the real-time dynamics of human conversation. For natural dialogue, the ability to search for external information in real-time is also an important capability. Many models remain trapped in parametric knowledge, leaving them unable to access real-time information and tool execution. Furthermore, even when Large Language Models (LLM) retrieve information, many duplex speech models process it within a compressed latent space rather than in its raw text form, which can lead to information loss from compression. To address this issue, we propose Context Spanning, a framework for information injection between a full-duplex speech model and an external LLM backend via real-time chunked prefill. The injected frame is encoded in a single forward pass inside the real-time frame budget. It feeds the retrieved information to the speech model as-is, enabling it to reason over the information independently and generate responses. With this approach, our model achieves high performance on Full-Duplex benchmarks and strong results on Question Answering tasks, demonstrating its conversation potential. Context Spanning shows that external information can be injected directly into a duplex speech model, introducing a new simple and powerful mechanism for duplex systems.
View PDFOpen arXiv