Training-free steering improves speech model task control with vectors
Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition
Computation and Language
Summary
Making speech recognition and understanding tools better usually requires retraining big models, which takes time and resources. This paper shows a way to guide these speech models to focus on specific tasks without retraining by adding special adjustment signals called steering vectors. These vectors are built from just a few examples and help the model perform tasks like transcribing speech or recognizing emotions more accurately. The approach works well even on new data types and can control the script used in different languages.
What this means in practice
- •For speech technology developers: Improve speech-to-text and emotion recognition accuracy by adding steering vectors during model use without retraining the whole model.
- •For multilingual app developers: Control and enforce specific written scripts for different languages easily within speech applications without retraining language models.
Authors
Séverin Baroudi, Yanis Labrak, Pierfrancesco Melucci, Sergio Burdisso, Petr Motlicek, Hervé Bredin, Mirco Ravanelli, Ricard Marxer
Abstract
Activation steering has proven effective for controlling the behavior of Large Language Models (LLMs) at inference time, but its application to SpeechLLMs remains new, and training-free steering approaches for such models are still largely unexplored. We propose a training-free Contrastive Activation Addition (CAA) protocol that derives steering vectors for common speech tasks (e.g. transcription) in SpeechLLMs from a small number of labeled utterances. We showcase that adding these vectors in the representation space, at inference time, enforces better the targeted speech task. We further show that, when combined with prompting, these vectors yield to consistent improvement over prompting alone on most evaluated tasks such as Automatic Speech Recognition (ASR) or Emotion Recognition (ER), and transfer to out-of-domain data. We additionally demonstrate the usefulness of script-normalization directions to enforce the target script of a specific language.