Speech models cut audio tokens by learning adaptive segmentation
ASSET: Adaptive Speech Segmentation via Reinforcement Learning
Computation and Language
Summary
Speech models often process long streams of audio data, making them slow and inefficient. The authors present a method called ASSET that learns to cut long audio into short meaningful pieces automatically, using feedback on task success and a penalty for too many pieces. This approach keeps the important information while reducing the number of pieces the model has to handle. Experiments show ASSET matches or improves on other methods for speech transcription while using fewer audio tokens. It also adapts segmentation styles when used for tasks beyond transcription.
What this means in practice
- •For speech recognition developers: Automatically create efficient audio input segments to improve transcription speed without sacrificing accuracy.
- •For voice assistant engineers: Reduce processing load in voice assistants by compressing audio inputs into fewer, informative tokens for better response times.
Authors
Cihan Xiao, Yuan Tseng, Matthew Wiesner, Desh Raj, Leibny Paola Garcia Perera, Sanjeev Khudanpur, David Harwath
Abstract
Speech large language models (LLMs) are often built using pretrained acoustic encoders as an audio interface to an LLM decoder. However, LLMs scale poorly with input sequence length, while acoustic encoders tend to produce substantially longer sequences than typical textual inputs. This motivates learning to compress encoder outputs while preserving task-oriented information. To this end, we introduce ASSET, a reinforcement learning (RL)-based framework that learns to adaptively compress encoder outputs by first partitioning them into variable-duration segments and then pooling each segment into a continuous audio token. We use group-relative policy optimization (GRPO) to train a segmenter with task performance feedback and a token-rate penalty. In automatic speech recognition (ASR) experiments, ASSET achieves performance comparable to a transcript-assisted character-boundary oracle with approximately 25% fewer audio tokens on English LibriSpeech, while improving on a phone-boundary oracle at a similar token rate on Mandarin AISHELL. Exploratory experiments on non-ASR tasks further illustrate how learned segmentation changes when the framework is adapted to tasks beyond transcription.