Voice recognition models evaluated against human perception of cloned voices

VIPBench: A Human-Aligned Benchmark for Voice Identity Perception in the Age of Voice Cloning

SoundArtificial IntelligenceComputers and SocietyMultimedia

Summary

Voice recognition tests usually check if a computer can identify who is speaking in a recording, but they don’t see if this matches what humans hear. The authors created a large test, VIPBench, where over a thousand people judged if pairs of real or computer-generated voices sounded like the same person. They found that humans often disagree with voice labels, especially for voice clones created by machines. The best computer models still don’t perfectly match human judgments, showing room for improvement by training models using human listening data.

What this means in practice

  • For voice security teams: Use VIPBench data to test and improve voice authentication systems against cloned or altered voices that may fool humans.
  • For voice assistant developers: Evaluate voice identity models to better align synthetic voices with how users perceive speaker similarity and identity.

Authors

Xudong Tang, Matthew Groh

Abstract

Speaker-recognition benchmarks test how accurately a model recovers the speaker label of a recording, treating voice identity as a property of the person who produced it. However, these benchmarks do not evaluate how humans perceive voice identity. Whether a listener hears two voices as the same person has consequences for intellectual property rights, cybersecurity, and everyday life. We introduce the Voice Identity Perception benchmark (VIPBench), which contains 124,876 same-or-different judgments from 1,290 listeners on 9,800 recording pairs from 100 public figures. The pairs include real recordings, voice clones generated by a text-to-speech system, and morphs between two speakers' voices. We also define four tasks that evaluate the alignment between speaker-representation models and human identity perception. The first two tasks measure how well a model's similarity score predicts the proportion of listeners who respond "same" and the majority response. The other two measure whether a model and listeners find the same speaker pairs similar, and whether a model's relationship to listener judgments holds equally for real and synthetic recordings. We evaluate eleven publicly available speech models. Listener judgments diverge from speaker labels most often for voice clones: in 41% of responses, listeners judged a clone and a real recording of the same speaker to be different people. No model reaches the noise ceiling; the best model explains 61% of the explainable variance. Models trained with speaker labels align with listeners better than self-supervised models do, yet a model with a 12% error rate on speaker labels aligns with listeners as well as models with error rates below 2%. VIPBench enables the evaluation of speaker models against human perception on real, cloned, and morphed speech, and our results motivate training speaker models directly on listener judgments.