Speech language models improve text accuracy with new training method
Reducing the Output-Mode Gap in Speech Language Models via Joint-Output On-Policy Distillation
Computation and LanguageSound
Summary
Speech language models that generate both text and speech together often produce less accurate text responses compared to generating text alone. The authors call this difference the output-mode gap. They introduce a training method that transfers knowledge from the better text-only generation to the combined text-and-speech mode. This reduces the accuracy gap while keeping overall model performance stable.
What this means in practice
- •For speech recognition developers: Improve accuracy of spoken answer systems that output both text and speech simultaneously for virtual assistants.
- •For voice assistant engineers: Enhance speech response quality in assistants by training models to better align joint text and audio output.$Commercial implications: Enables creation of more accurate and natural-sounding voice assistants with improved spoken answer reliability for consumers.
Authors
Daxin Tan, Dehua Tao, Chengxi Deng, Hanlin Zhang, Xiao Chen
Abstract
Autoregressive generation of interleaved text and acoustic tokens is a common approach to spoken-response generation in speech large language models. Although this design enables streaming generation with explicit textual guidance, generated acoustic tokens become part of the context for subsequent text predictions. Given identical speech inputs, we observe markedly lower answer accuracy for the internal text generated in speech-to-text-and-speech (S2TS) mode than for speech-to-text (S2T) responses. We term this discrepancy the \emph{output-mode gap} (OMG). To reduce OMG, we propose \emph{Joint-Output On-Policy Distillation} (JO-OPD), which distills the model's stronger S2T policy into joint generation using student-generated S2TS trajectories. At each text position, the S2T teacher provides soft targets from a text-only projection of the student's preceding outputs, while the student predicts from the corresponding full interleaved history. A preservation objective further regularizes native non-text predictions. Experiments on Step-Audio-2-mini and Baichuan-Audio-Instruct reveal OMG across two interleaved generation architectures. On Step-Audio-2-mini, JO-OPD reduces OMG from 42.87 to 16.26 percentage points on Spoken-MQA and from 29.72 to 13.04 points on speech-rendered GSM8K, with little change in S2T accuracy and substantially larger reductions than matched SFT baselines. ASR-based evaluation further shows a 7.49-point improvement in spoken-answer accuracy on Spoken-MQA.