Neural audio codecs reveal and alter speaker traits with sparse codes
Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: Exploration toward Age, Gender, and Accent Steering
Sound
Summary
Audio compression tools called neural audio codecs store voice information in ways that are hard to interpret. The authors studied how these tools capture traits like age, gender, and accent by using a special method to find parts of the code linked to these traits. Changing those parts can shift the perceived speaker traits, but it often makes the speech sound worse and less clear. This shows that while the codes do hold speaker details, separating them cleanly from other audio features is still a challenge.
What this means in practice
- •For voice technology developers: Identify and manipulate speaker traits in compressed speech to enhance voice personalization features.
- •For language learning app makers: Improve feedback by detecting and altering accents in speech samples for more tailored pronunciation training.
Authors
Shih-Heng Wang, Tiantian Feng, Aditya Kommineni, Huang-Cheng Chou, Bowen Yi, Xuan Shi, Shrikanth Narayanan
Abstract
Neural audio codecs (NACs) are widely used in speech generation and audio-language modeling, yet how they encode speaker-trait information remains poorly understood. Prior work applied sparse autoencoders (SAEs) to investigate accent information in NACs through task-level analysis. Here, we extend this analysis to the waveform level and to age, gender, and accent, using SAE steering to probe trait-related information in sparse activations. We identify trait-associated dimensions, modify their activations, and evaluate the resulting reconstructed speech. Across five NACs, steering the selected dimensions induces target-directed shifts in speaker-trait predictions. A random-dimension baseline on Mimi produces smaller shifts, supporting the relevance of the selected dimensions. However, responses vary across codecs, traits, and steering directions, and increasing steering strength does not consistently amplify the intended shifts. Steering also generally increases word error rates and lowers predicted perceptual quality. These findings suggest that SAEs capture speaker-trait information in steerable activations, while the accompanying quality degradation highlights the need to better separate trait-related information from other information.