Speech to language models framework improves generalization beyond transcription

DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs

Computation and Language

Summary

Speech-based language models usually learn only to transcribe speech but struggle with new tasks like translating speech or recognizing emotions. The authors created a simple method called DirectSpeech2LLM that helps a language model understand and follow new instructions when given speech input. By using a special training loss and aligning speech features with model embeddings, their system learns from just transcription data but works well on unseen tasks. This method performs better than combined systems and gets close to their top performance on tasks the model never saw during training. The approach works across multiple language model types and improves as it uses more data or bigger models.

What this means in practice

  • For speech recognition developers: Build speech-to-text systems that maintain performance while enabling generalization to speech translation and emotion recognition without retraining.
  • For multilingual technology teams: Create speech translation tools that extend beyond transcription using models trained only on speech recognition data.$Commercial implications: Supports new speech translation products without collecting task-specific training data, reducing development cost and time.

Authors

Hemant Yadav, Sunayana Sitaram, Roger Zimmermann, Rajiv Ratn Shah

Abstract

Speech-LLMs often exhibit prompt overfitting, where models solely trained on automatic speech recognition (ASR) instruction fail to generalize to new instructions such as speech translation and continue to behave primarily as ASR system. We propose DirectSpeech2LLM, a simple end-to-end framework that preserves the instruction-following ability of the LLM on unseen tasks when conditioned on speech. It computes distance-based CTC loss over the frozen LLM embedding matrix and uses greedy CTC labels to derive geometrically and temporally aligned speech embeddings respectively as an input to the LLM. Trained solely on 960 hours of LibriSpeech ASR data, DirectSpeech2LLM outperforms the cascaded system on ASR (seen task) and generalizes zero-shot to speech translation and emotion recognition (two unseen tasks), closely matching the cascaded system upper bound on these two new instructions despite seeing neither during training. We also find that geometric alignment strength plays a smaller role than previously assumed, as our modified CTC loss is shown to provide sufficient implicit geometric grounding without requiring an explicit regression loss. Results are consistent across two LLM families and scale with both more training data and model capacity.
View PDFOpen arXiv