Speech language models improve reasoning accuracy with real time self correction
RetroThinker: Enabling Retrospective Thinking in Speech LLMs
Artificial IntelligenceComputation and Language
Summary
Speech language models can understand spoken words faster and keep voice details better than converting speech to text first. But they are not as good as text-based models at solving tricky problems quickly. The authors created RetroThinker, which lets a speech model check and fix its own thinking steps while listening. This approach helps the model be more accurate without slowing it down much. Tests showed RetroThinker improved problem-solving scores by 11% with similar speed.
What this means in practice
- •For voice assistant developers: Enable voice assistants to improve reasoning steps on the fly while the user speaks, boosting accuracy without adding delay.$Commercial implications: RetroThinker allows production voice assistants to provide more accurate real-time spoken responses with self-corrected reasoning.
- •For customer support automation teams: Enhance automated phone support systems to better understand and respond to complex queries through dynamic reasoning corrections.
Authors
Yi-Jen Shih, Puyuan Peng, Abdelrahman Mohamed, David Harwath
Abstract
Speech large language models (SpeechLLMs) offer reduced latency and retain paralinguistic nuances that are typically lost in cascaded automatic speech recognition (ASR) and text-based LM architectures. However, they continue to lag behind text-only LLMs on complex reasoning tasks, while real-time spoken interaction imposes strict latency constraints. Although prior works employ Chain-of-Thought (CoT) and concurrent reasoning to enhance reasoning capabilities without inducing prohibitive delays, an inherent accuracy-latency trade-off persists. In this paper, we investigate whether a streaming SpeechLLM can dynamically revise its reasoning traces on the fly. We introduce RetroThinker, a multi-stage post-training framework that equips the Moshi model to self-verify and forward-correct CoT steps during inference. RetroThinker combines supervised fine-tuning (SFT) on curated retrospective thinking data with length-based direct preference optimization (DPO) to optimize retrospective during early reasoning (i.e., reasoning concurrently while the user speaks). Evaluated on the GSM8K benchmark, RetroThinker significantly improves the accuracy-latency trade-off over non-retrospective baselines, achieving an 11% absolute accuracy gain at a comparable latency.