X-AuT cuts audio encoder layers to improve speech model efficiency
X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
SoundArtificial Intelligence
Summary
Large speech language models use many layers to understand and process audio, but running all these layers can be slow and costly. The authors propose X-AuT, a method that carefully removes some layers without hurting the model’s ability to understand speech. They do this by teaching the smaller model to mimic the bigger one through a step-by-step process. Their approach keeps the model accurate while making it faster and using fewer resources.
What this means in practice
- •For speech recognition system engineers: Reduce computational cost and latency in speech recognition models by selectively compressing audio encoders without significant loss in accuracy.
- •For mobile app developers: Develop lighter speech-enabled apps that run efficiently on limited hardware by applying progressive audio-encoder compression techniques.
Authors
Haojun Zhang, Yi Zou, Min Chen, Qize Yu, Lianrui Fan, Xini Ding, Hao Li, Shuchang Zhou, Xianming Liu, Shiyu Huang
Abstract
Reducing audio-encoder depth lowers the inference cost of speech large language models, but removing complete blocks perturbs the embeddings consumed by the decoder and can cause deletion and premature end-of-sequence errors. We introduce X-AuT, a progressive framework that selects layer combinations through short behavioral probes and restores the pruned model through representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning. The language-model backbone remains frozen, while attention LoRA adapters and the tied output embedding adapt during distillation. Training uses the highest-agreement tier from a transcript-consistency pipeline, followed by source reweighting during finetuning. On ten public Chinese--English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers reduces macro-average error from 5.61% to 5.27%. The 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters. Under the matched recipe, the 1.7B teacher yields 5.55% mean error, compared with 8.45% for self-distillation, and progressive 18$\rightarrow$14 pruning outperforms direct pruning (5.75% vs. 6.73%). These single-run results establish two practical operating points and show that the accuracy effects vary across benchmarks. Project website: https://xpeng-ai.github.io/x-aut