Speech conversation benchmark reveals challenges in multi-turn math reasoning
SCB: SpeechConversationBench for Evaluating Multi-Turn Reasoning in Speech-to-Speech Models
Computation and LanguageArtificial IntelligenceSound
Summary
Speech-to-speech systems must handle problems that unfold over several conversational turns, like solving math questions step-by-step by listening to each part. The authors designed a test called SpeechConversationBench that checks how well these systems perform math reasoning when problems are given all at once or broken into parts across turns. They found that commercial systems drop accuracy when information is split across turns, but their own LEGO system handles incremental spoken information better, achieving high accuracy. This shows that managing conversational context explicitly helps speech systems reason across turns more effectively.
What this means in practice
- •For voice assistant developers: Improve dialogue management in voice assistants to better handle math or logical tasks presented over multiple conversational turns.
- •For customer service chatbot teams: Enhance interactive speech-based chatbots to track and integrate incremental user inputs for more accurate problem solving during phone conversations.
Authors
Kanpat Vesessook, Saksorn Ruangtanusak
Abstract
Speech-to-speech systems must solve tasks whose requirements emerge across conversational turns. We introduce SpeechConversationBench (SCB), a focused evaluation of spoken mathematical reasoning using 103 sharded GSM8K problems. The framework compares the original problem delivered in one turn (full), its concatenated information shards delivered together (concat), and incremental spoken disclosure across turns (sharded). We report final-answer accuracy for four commercial speech systems and LEGO, a proprietary speech pipeline developed internally by the SCBX Innovation Lab team with explicit conversational context management. Relative to concat, sharded accuracy decreases by 5.0-25.3 percentage points across the four commercial systems. LEGO achieves 77.5 percent accuracy in all three conditions, compared with 76.6 percent sharded accuracy for GPT-4o Realtime. The two single-turn baselines distinguish sensitivity to problem reformulation from the additional challenges introduced by incremental spoken interaction.