Speech models balance recognition and question answering with new training method
TS-OPD: Reconciling ASR and QA in Speech Language Models via Task-Specific On-Policy Distillation
Sound
Summary
Speech language models can either be good at understanding spoken words (ASR) or good at answering questions (QA), but improving one often hurts the other. The authors propose a new training method called TS-OPD that uses two versions of a model, one focused on ASR and one on QA, to teach a single model both skills without conflict. This method helps the model recognize speech better while still answering questions well. The approach works well across different tasks and training settings.
What this means in practice
- •For voice assistant developers: Create speech-enabled assistants that recognize spoken commands accurately while also answering user questions without loss in performance.
- •For call center software teams: Improve systems that transcribe customer calls and provide relevant answers simultaneously, maintaining quality in both tasks.
Authors
Yujie Guo, Hongjie Chen, Jian Kang, Jie Li, Yongxiang Li, Yong Qin
Abstract
Speech Language Models (SLMs) inherit strong instruction-following capabilities from pretrained language models, yet ASR specialization can substantially degrade them. To address this ASR--QA trade-off, we propose Task-Specific On-Policy Distillation (TS-OPD), which leverages models before and after ASR specialization as complementary QA and ASR teachers. The student generates separate task-conditioned trajectories for ASR and QA, each supervised only by its corresponding teacher, thereby reducing direct competition between the two supervision signals. Experiments on basic ASR, contextual ASR, and QA demonstrate that TS-OPD improves recognition while preserving QA capability. Moreover, TS-OPD remains robust across different balancing coefficients and continues to benefit from increased distillation data.