Spoken language models struggle to understand long audio conversations
Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models
Computation and LanguageHuman-Computer Interaction
Summary
Long spoken conversations are hard for language models to understand because the models forget old information as the conversation goes on. This problem is even bigger with speech than text since audio needs more data to keep both meaning and sounds. The researchers created Vox-Infinity, a new test designed to see how well models remember long audio conversations by extending the length of past speech in different ways. They tested seven models and found they do better when relevant information is recent but have trouble using information from further back in the conversation.
What this means in practice
- •For speech technology developers: Improve spoken language models by evaluating and targeting their ability to use long audio histories effectively.
- •For customer service platform teams: Measure and enhance virtual assistants’ understanding of extended phone calls or conversations with multiple exchanges using the benchmark.$Commercial implications: Enables building and selling virtual assistants better at handling long calls, improving customer satisfaction and operational efficiency.
Authors
Xize Cheng, Wenxu Jia, Chenyuhao Wen, Dongjie Fu, Zehan Wang, Xinyu Zhang, Tao Jin
Abstract
Long-context understanding remains a fundamental challenge for large language models, as excessively long inputs often lead models to forget salient information. This issue is even more pronounced in the speech domain, where audio, as a low-compression modality, requires substantially more embeddings than text to preserve both semantic content and acoustic cues. To address this challenge, we introduce \textbf{Vox-Infinity}, the first benchmark specifically designed to evaluate long-context understanding in spoken language models. Vox-Infinity systematically extends audio history along two dimensions: turn count and turn duration. It covers a diverse range of representative scenarios with varying interaction structures and semantic complexity. Crucially, Vox-Infinity provides explicit answer-provenance annotations and organizes samples according to the amount of historical context required to resolve each query, enabling precise and length-aware evaluation. Extensive evaluations of seven representative spoken language models reveal a clear overall recency effect: models generally achieve higher accuracy when answer-supporting evidence is closer to the query, but struggle to retrieve and use evidence located farther back in the dialogue history. Cases and datasets are available at https://vox-infinity.github.io.