Efficient speech enhancement models found using reinforcement learning
Efficient Audio-Visual Speech Enhancement via Reinforcement Learning-Based Neural Architecture Search
SoundArtificial Intelligence
Summary
Improving speech clarity in noisy places is hard on small devices because the best models are often too big and slow. The authors used a type of AI called reinforcement learning to design smaller, faster audio-visual speech enhancement models that still work almost as well as larger ones. Their approach tests fewer model options and finds designs that keep good speech quality while being easier to run on phones. This method could help make smart noise-clearing technology more practical for everyday use.
What this means in practice
- •For mobile app developers: Build speech enhancement apps that run efficiently on mobile phones by using smaller models found through reinforcement learning.
- •For hearing aid manufacturers: Incorporate compact audio-visual speech enhancement models into hearing aids to improve users’ speech clarity in noisy environments.$Commercial implications: This paper enables the creation of lightweight speech enhancement models suitable for embedded hearing aid devices, improving real-world usability.
Authors
Khubaib Ahmed, Ahsan Adeel, Nasir Saleem, Kia Dashtipour, Ahsan Ulhaq, Amir Hussain
Abstract
Audio-visual speech enhancement (AVSE) improves speech intelligibility in noisy environments but often relies on computationally expensive models unsuitable for resource-constrained edge devices. This paper introduces a reinforcement learning-driven neural architecture search (RL-NAS) framework to discover compact, efficient AVSE architectures. The search space preserves temporal alignment between audio and visual streams, while a novel reward function jointly optimises speech enhancement performance, measured by SI-SNR improvement, and model compactness. Using proximal policy optimisation with constraint-aware action masking, the framework evaluates only 35 candidate architectures and identifies a model with 2.9 million parameters, compared with 22 million in the baseline. The proposed architecture achieves a 14.5 dB SI-SNR improvement, only 0.7 dB below the baseline, while reducing model size by approximately nine-fold. Evaluation on commercial mobile CPUs demonstrates a 4.3-fold reduction in real-time factor, supporting practical on-device deployment. Compared with random search, RL-NAS achieves 92% higher search efficiency. Architectural analysis reveals that shallower but wider temporal convolutional blocks and aggressive visual pathway pruning can maintain enhancement performance while substantially reducing computational requirements. These findings demonstrate the potential of reinforcement learning-guided architecture search for developing lightweight, efficient and deployable multimodal speech enhancement systems.