Speech models improve thinking speed and accuracy with dynamic self-correction
RetroThinker: Enabling Retrospective Thinking in Speech LLMs
Artificial IntelligenceComputation and Language
Summary
Speech-based language models can understand spoken words with less delay and capture tone better than systems that first convert speech to text. But they struggle with tricky reasoning tasks while also needing to respond quickly. The researchers created RetroThinker, which helps these models double-check and fix their own reasoning as they listen and talk. This approach boosts their accuracy by 11% without making them slower. It shows that letting models rethink their answers on the fly can improve how well they understand complex speech.
Speech large language modelsAutomatic speech recognitionChain-of-Thought reasoningLatencyFine-tuningInferenceRetrospective thinkingGSM8K benchmarkDirect preference optimizationParalinguistic nuances
Authors
Yi-Jen Shih, Puyuan Peng, Abdelrahman Mohamed, David Harwath
Abstract
Speech large language models (SpeechLLMs) offer reduced latency and retain paralinguistic nuances that are typically lost in cascaded automatic speech recognition (ASR) and text-based LM architectures. However, they continue to lag behind text-only LLMs on complex reasoning tasks, while real-time spoken interaction imposes strict latency constraints. Although prior works employ Chain-of-Thought (CoT) and concurrent reasoning to enhance reasoning capabilities without inducing prohibitive delays, an inherent accuracy-latency trade-off persists. In this paper, we investigate whether a streaming SpeechLLM can dynamically revise its reasoning traces on the fly. We introduce RetroThinker, a multi-stage post-training framework that equips the Moshi model to self-verify and forward-correct CoT steps during inference. RetroThinker combines supervised fine-tuning (SFT) on curated retrospective thinking data with length-based direct preference optimization (DPO) to optimize retrospective during early reasoning (i.e., reasoning concurrently while the user speaks). Evaluated on the GSM8K benchmark, RetroThinker significantly improves the accuracy-latency trade-off over non-retrospective baselines, achieving an 11% absolute accuracy gain at a comparable latency.