Summary
Speaker verification systems are better at telling who is speaking when they use large speech models, but adjusting these models fully can be costly. The authors introduce a method that changes only small parts of the model to improve performance efficiently. They separate the model’s internal signals into groups and carefully mix them using a simple control system that gradually learns how to adapt. They also create a way to weigh the importance of different layers in the model based on each moment of speech, which helps the system focus on the right details. Their approach improves accuracy significantly while only requiring very small changes to the original model.
What this means in practice
- •For voice biometric system developers: Improve speaker verification accuracy using small, efficient model adaptations that require very little retraining on new data.
- •For call center technology teams: Integrate adaptive speaker verification that adjusts dynamically to varying speech frames, enhancing identity checks in real time.
Authors
Zezhong Jin, Xiaoyu Wang, Zhe Li, Chong-Xin Gan, Siqing Qin, Man-Wai Mak, Kong Aik Lee
Abstract
Large self-supervised speech models provide powerful representations for speaker verification, but full fine-tuning is expensive. Most conventional adapters insert small trainable modules to modify the output of each layer, while keeping the original residual connections unchanged. Building on our preliminary study of manifold-constrained Hyper-Connections in speaker encoders, we explore parameter-efficient adaptation of both residual paths and feature selection across layers. We first propose a Manifold-Constrained Residual Adapter (MCRA), which splits the hidden dimensions of each Transformer residual state into N groups and uses a small Sinkhorn-normalized matrix to exchange information among them. A learnable gate controls the balance between the original and mixed residual paths. The gate adds only one scalar parameter at each residual connection and initially favors the original path, allowing gradual adaptation. We further propose Frame-Adaptive Layer Aggregation (FALA), an attention-based aggregation method that uses a single learned query shared across all utterances to compute layer weights separately for each speech frame. Because the layer representations depend on the input, the resulting weights vary across both utterances and frames, unlike the fixed weights of a conventional layer-weighted sum. Compared with a frozen WavLM baseline using a fixed layer-weighted sum, our method achieves relative EER reductions of 22.4% on VoxCeleb1-O and 2.9% on CN-Celeb, with trainable adaptation parameters equivalent to only 0.0009% and 0.0016% of the WavLM-Large parameters, respectively.