Audio models learn to verify who is speaking more accurately
TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models
Sound
Summary
Many AI systems can understand what is being said but have trouble telling who is speaking. The authors developed a two-step method that helps these models keep information about the speaker's identity and use it to recognize voices better. They tested their method on a popular audio language model and found it reduced errors in speaker verification. This means the improved model can tell if two speech samples come from the same person more reliably.
What this means in practice
- •For voice assistant developers: Improve voice authentication by enabling AI models to better verify user identity during audio interactions.$Commercial implications: This method makes commercial voice recognition systems more secure by enhancing speaker verification accuracy.
- •For security software teams: Build more reliable speaker verification features for access control systems using enhanced speaker-preserving audio representations.
Authors
Junjie Li, Zheng Liang, Zhe Li, Tianchi Liu, Kong Aik Lee
Abstract
Audio large language models (ALLMs) can understand speech content, yet their ability to use speaker identity for verification remains limited. We propose TS-SP (Two-Stage Speaker Preservation), a parameter-efficient framework for learning speaker-preserving representations and making them accessible to an ALLM's language-model component. We instantiate and evaluate TS-SP on Qwen2.5-Omni-7B. First, we adapt the audio encoder with speaker identity supervision. We then freeze the adapted encoder and train the language model to compare speakers. Both stages use low-rank adaptation (LoRA), keeping the pretrained base weights fixed. On Vox1-O, TS-SP reduces the equal error rate (EER) from 7.01\% for the Paired Loss Adaptation Baseline to 4.37\%. EER remains within 4.31--4.79\% under unseen prompts. Cross-domain evaluation on CN-Celeb yields a similar EER to the baseline, but lower accuracy at the native decision threshold. These findings support two-stage adaptation for improving speaker verification on the evaluated backbone.