Voice similarity better measured by shape of embeddings than accuracy
Rethinking Automated Voice Similarity by Shifting from EER to Embedding Geometry
Sound
Summary
People often assume that better speaker verification accuracy means a computer is better at judging if two voices sound alike. The authors found that how the model is trained matters more than accuracy for matching human opinion. They showed that certain training methods produce voice representations that align better with how humans perceive voice similarity. This happens because human judgment is based on fewer key voice features, while some models spread information across many dimensions. By making the model focus on fewer features, it matches human voice similarity much better.
What this means in practice
- •For voice assistant developers: Improve voice generation quality by selecting models whose embeddings better match human perception of voice similarity.
- •For audio forensics teams: Use embedding geometry as a reliable measure to assess whether two voice samples sound alike to humans.
Authors
Szu-Chi Chen, Jia-Kai Dong, Yi-Cheng Lin, Sung-Feng Huang, Hung-yi Lee
Abstract
Speaker verification (SV) models are commonly assumed to better capture nuances among speaker characteristics as verification accuracy improves, leading to their widespread use as automated proxies for human voice similarity in speech generation tasks. However, by establishing a human perceptual alignment metric and conducting systematic analysis, we demonstrate that perceptual alignment is governed far more by how a model is trained (its learning objective) than by how well it performs (EER). Notably, standard margin-based classification losses (e.g., AAM-Softmax) yield substantially lower perceptual alignment than prototypical metric losses, while EER itself fails to track human judgment, directly challenging the community's implicit assumption. We trace this divergence to embedding geometry, where a model's effective dimensionality ($d_{\mathrm{eff}}$) tracks perceptual alignment with a $-0.95$ rank correlation, revealing that the dimensional spread favored by classification losses fundamentally clashes with the low-dimensional nature of human voice perception. Imposing a dimensionality bottleneck compresses $d_{\mathrm{eff}}$ and raises perceptual alignment ($ρ_{\mathrm{align}}$) from 0.08 to 0.74, establishing a principled geometric criterion for evaluating voice similarity.