Neural codec compression weakens audio deepfake detectors performance
Exposing and Mitigating Neural Codec Vulnerabilities in Audio Deepfake Detection
Sound
Summary
Audio deepfake detectors, which aim to spot fake speech, usually handle traditional audio changes like adding noise or simple compressions. The authors found that new speech synthesis methods use neural codecs that create different audio artifacts. These detectors struggle when audio is compressed or resynthesized with such neural codecs, sometimes mistaking real compressed speech for fake. To fix this, the authors made a new dataset and proposed a method called PCL-NET that trains detectors to recognize speech correctly even after neural codec compression, improving detection accuracy.
What this means in practice
- •For audio security teams: Improve detection systems to correctly identify deepfake audio even after neural codec compression, reducing false alarms on compressed legitimate speech.
- •For telecommunications engineers: Design systems that maintain speech authenticity verification through neural codec compression used in low-bandwidth communication.
Authors
Abdullah, Awais Khan, Khalid Mahmood Malik
Abstract
Existing audio deepfake detection (ADD) datasets and detectors are primarily built for vocoder-based synthesis, evaluated against traditional post-hoc perturbations such as MP3/AAC compression or additive noise, applied independently of generation. However, recent speech synthesizers, particularly ALM-based systems, use neural audio codecs both for compression and as the resynthesis reconstructing waveforms from generated tokens, producing artifacts distinct from post-hoc compression. Neural codecs thus play a dual role: some are designed for pure compression under low-bandwidth communication, while others serve as resynthesis components. Despite this dual role, robustness to codec-based compression, unlike post-hoc compression, remains largely unexplored. We expose this gap, showing that state-of-the-art (SOTA) ADD models degrade drastically on codec-compressed speech; in particular, systems trained on Codec Resynthesized data as a proxy for codec-based generation prove most vulnerable, with legitimately compressed bona fide speech often misclassified as fake. To investigate this, we construct the Audio Neural Codec-Spoof dataset by applying seven neural codec algorithms to existing ADD benchmarks, isolating codec-induced resynthesis artifacts as a controlled proxy for codec-based generation. As baseline mitigation, we propose PCL-NET (Pairwise Consistency Learned Network), fine-tuning a pretrained XLS-R (300M) encoder with a pairwise consistency objective that minimizes the representation distance between an utterance's uncompressed and codec-compressed versions, disentangling codec artifacts from the real-versus-fake decision. As a result, PCL-NET reduces average EER under neural codec compression from 28.67% to 12.77%, while preserving competitive CoSG-based deepfake detection performance. We will also make the dataset publicly available on Hugging Face upon acceptance.