AI summaryⓘ
The authors studied Spoken Language Models (SLMs), which turn speech directly into text responses, bypassing traditional step-by-step systems. They found that current SLMs don't follow instructions as well or handle many tasks as effectively as text-only language models, partly because speech and text are represented differently inside the models. To fix this, the authors created a new method that separates the problem of different input lengths from how speech and text meanings align, helping the model better connect spoken sounds to written words. Their experiments show this approach works well compared to others, highlighting the importance of addressing structural differences between speech and text when training SLMs.
Spoken Language Modelsspeech representationtext representationsemantic alignmentinstruction-followingcontinuous speechdiscrete textlength mismatchmodel generalizationSLM training
Authors
Hyeonyu Kim, Hwayeon Kim, Youngwon Choi, Myeongkyun Cho, Huu-Kim Nguyen
Abstract
Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.