Speaker privacy needs more than one way to measure leakage
Beyond EER: Multi-Dimensional Evaluation of Information Leakage in Speaker De-Identification
SoundArtificial Intelligence
Summary
Protecting who is speaking in recordings is important to keep people’s identities private. The authors found that current methods usually check this by measuring one thing called Equal Error Rate, which isn’t enough to catch all ways information can leak out. They suggest checking five different ways that privacy might be at risk, including how much personal info can be guessed and how similar disguised voices still look to computers. Testing this on five systems showed that relying on just one measurement can miss important privacy problems.
What this means in practice
- •For privacy engineers: Evaluate speaker masking systems with multiple privacy metrics to ensure stronger protection against identity leaks.
- •For voice assistant developers: Improve voice anonymization methods by testing them beyond basic accuracy measures to protect user identity more effectively.
Authors
Seungmin Seo, Oleg Aulov, P. Jonathon Phillips, Kevin Mangold, Jonathan Eskin
Abstract
Speaker de-identification (SDID) aims to preserve privacy by concealing speaker identity while maintaining speech utility. However, current evaluations often reduce privacy to a single dimension - biometric verification performance - typically measured by Equal Error Rate (EER). This narrow focus ignores critical leakage channels, such as soft biometric inference, embedding-level re-identification, and structural template similarity, which threaten the unlinkability and irreversibility of biometric references. We propose a holistic evaluation framework across five complementary metrics: (i) EER, (ii) soft biometric leakage score , (iii) cumulative match characteristic re-identification analysis, (iv) canonical correlation analysis and Procrustes embedding alignment, and (v) intelligibility via word error rate and semantic similarity. Evaluating five SDID systems from the IARPA ARTS program, we demonstrate that these metrics capture independent dimensions of information leakage. Our results indicate that reliance on a single metric can misrepresent the privacy properties of an SDID system.