SlAM-asr shows limited spoken understanding from speech transcription features
Zero-Shot Spoken Language Understanding with SLAM-ASR: What Do Monolingual ASR-Trained Linear Projectors Enable?
Sound
Summary
This work studies whether a speech transcription method called SLAM-ASR can also understand the meaning of spoken language without extra training. The authors tested this on 11 languages by training simple language-specific adapters and asking yes/no or topic questions about the speech. They found the system could follow instructions about half the time for yes/no questions and better (over 80%) when given multiple choice options. However, the method did not work as well as when given the actual transcript text, showing that understanding spoken language from raw speech features remains a challenge.
What this means in practice
- •For speech technology teams: Enable zero-shot evaluation of spoken language understanding on new languages using frozen ASR encoders with monolingual projectors.
- •For multilingual ai engineers: Assess whether linear projectors trained on monolingual transcription data support downstream spoken language understanding tasks without retraining the language model.
Authors
Edwin Arkel Rios, Sneh Sourabh, Jocelyn Zaruma, Girish Ewoorkar, Julian Mack, Bo-Cheng Lai
Abstract
The SLAM-ASR framework connects the features of a frozen speech encoder as input to a frozen LLM decoder to enable speech transcription. In this work we ask whether this training-on speech transcription-provides the LLM with the capability to also perform spoken language understanding (SLU) tasks, by training monolingual projectors on 11 languages. When prompted to answer a yes/no question about the input speech, the mean instruction following rate (IFR) is 49.8%. When not following the instruction, these models default back to transcription. The transcription quality-measured by the character error rate (CER)-is not sufficient to predict IFR. Remarkably, when the model is instructed to answer the topic question as a single digit option, the mean IFR rises to 81.3%. Our models score a mean 23.2% topic balanced accuracy (BAcc) against 14.3% chance. However, the same decoder scores a mean 71.5% BAcc when using reference transcripts as text input. This indicates a failure of the projector to transfer information for the SLU tasks.