Sharing layers enables smaller speech models for edge devices
Depth through Recurrence: Towards Ultra-Efficient On-Device ASR
Machine Learning
Summary
Speech recognition usually requires big models that use a lot of memory and battery, which is a problem for small devices like watches and glasses. The authors noticed many layers in deep speech models do very similar things, so they created a new design that reuses the same layers multiple times instead of having many unique ones. This makes the model smaller and more efficient without losing much accuracy. Their approach also allows easy personalization for individual users with low energy cost.
What this means in practice
- •For mobile device developers: Build compact speech recognition models that run efficiently on devices with limited memory and battery.
- •For smartwatch manufacturers: Deploy personalized speech recognition that adapts to users without increasing computational cost or energy use.$Commercial implications: Enables smartwatches to offer better voice interaction with personalized models running efficiently on-device.
Authors
Chen Feng, Tianyi Xu, Ramchalam Kinattinkara Ramakrishnan, Yicheng Lin, Shaojie Zhuo, Zhaocong Yuan, Chenzheng Su, Xiaopeng Zhang
Abstract
Modern automatic speech recognition (ASR) systems have achieved remarkable accuracy by scaling model depth and capacity, but at the cost of increased memory, energy, and computational demands. On edge devices where ASR is often most needed, such as watches and glasses, deploying such large models is infeasible due to extremely constrained resources. This raises a fundamental question: can we achieve high representational power in deep ASR models without scaling up parameterization? In this work, we revisit the role of depth and identify layer wise representational dynamics, in which most layers learn functionally similar transformations. Motivated by this insight, we propose a block-recurrent ASR architecture that replaces parameterized depth with a small set of recurrent blocks, each consisting of weight-shared layers, thereby preserving effective depth while drastically reducing model size. Through representation-guided grouping and knowledge distillation, block-recurrent models retain the accuracy of large reference models while using an order of magnitude fewer parameters. Across Open ASR Leaderboard benchmarks, a model with only two shared blocks recovers 97.4% of the accuracy of large models on average. The compact backbone further enables efficient on-device personalization through lightweight MoE-LoRA adaptation, allowing user-specific models to match or surpass the accuracy of general-purpose ASR systems with significantly lower inference cost. By reusing weights across recurrent steps, the proposed architecture reduces memory-access overhead and inference energy consumption. Together, these results show that high quality ASR can be achieved without large parameterization, and that structured recurrence provides a principled path towards ultra-compressed, energy-efficient, and personalized speech recognition on the edge.