Speech lLM pruning improved by component specific layer scoring

Speech Block Influence: Component-Specific Layer Scoring for Pruning Speech LLMs

Sound

Summary

Speech large language models (LLMs) are often too big and slow for devices with limited resources. The authors found that existing ways to decide which parts of these models to cut don’t work well because speech LLMs process both audio and text in complicated ways. They created a new method called Speech Block Influence that scores different parts of the model separately based on their roles, making it easier and safer to trim the model without hurting its performance. Their method also shows that focusing only on text parts might be enough to figure out which layers are important, saving effort.

What this means in practice

  • For speech technology engineers: Reduce computational cost of speech LLMs by pruning models more effectively using component-specific layer importance scores.
  • For mobile app developers: Deploy efficient speech LLMs on devices with limited resources by trimming unnecessary layers identified by the proposed scoring method.

Authors

Siyu Yao, Du Q. Huynh, Lian Xu, Mark Reynolds

Abstract

Speech LLMs are costly to deploy in resource-constrained settings. Layer pruning can cut this cost, but existing scoring metrics transfer poorly to speech LLMs: they assume a decoder-only architecture with homogeneous token sequences, whereas speech LLMs add encoder and adapter components and process multimodal sequences. We propose Speech Block Influence (SBI), the first layer-importance scoring framework designed for speech LLM pruning that consists of two component-specific scores: SBI-Enc measures the effect of encoder-layer removal at the adapter's output to better reflect downstream impact; SBI-Dec measures layer-wise input-output similarity over text-token positions only to avoid audio-token dominance. Across three speech LLMs, SBI improves pruning robustness, with stronger encoder performance at higher pruning rates and more reliable decoder layer selection by scoring text tokens rather than the audio-dominated full sequence. We further find that text-only calibration yields decoder rankings highly correlated with those from speech-text calibration, suggesting a cheaper alternative to measure decoder layer importance.