Do Speech Tokens Leak Voiceprints? Speaker Inversion Attacks Against End-to-End Speech Language Models
SoundArtificial IntelligenceCryptography and Security
Summary
The authors study if speech tokens, which are digital representations of spoken words used in speech language models, can reveal a speaker's unique voice. They call this risk a 'speaker inversion attack,' where someone tries to reconstruct a person's voice from these tokens. To explore this, they created a new model named Audio BERT (AuB) and a method called SpInv that attempts to recover speaker-specific voice features. Their tests show that SpInv can successfully guess a speaker's voice characteristics with only a few seconds of speech token data. This suggests speech tokens might leak sensitive speaker information.
Authors
Ye Lu, Yihan Yan, Zhaoyang Zhang, Zhitao Ou, Runze Liu, Li Liu, Shen Wang
Abstract
End-to-end speech language models increasingly represent user speech with speech tokens rather than relying exclusively on cascaded ASR--LLM--TTS pipelines. Although these tokens support expressive and low-latency spoken interaction, they may also preserve sensitive speaker characteristics. We investigate whether exposed speech tokens leak voiceprints and formulate this risk as a speaker inversion attack. We introduce Audio BERT (AuB), a trainable model that constructs token embeddings from discrete codebooks and aggregates them into speaker-sensitive representations, and propose SpInv, a two-stage inversion method built on AuB to recover embeddings in the space of an attacker-specified speaker encoder. We evaluate Moshi, Higgs3, Kimi-Audio, and Qwen3-Omni using speaker-disjoint protocols on the VoxCeleb dataset. Extensive experiments show that, with only three seconds of frontend output, SpInv achieves cosine similarities above 0.70 in the attacker-specified speaker-encoder space.