Deepfake speech detection improved by fusing complementary audio models
CRAF: Cross-View Residual-Aware Fusion for Deepfake Speech Detection
SoundArtificial Intelligence
Summary
Deepfake speech can sound very real, making it hard to detect fake voices that impersonate people. The authors developed a new method called CRAF that combines two types of sound analysis models: one that focuses on fine details in the audio and another that understands higher-level speech context. By carefully mixing these different views, the system can better spot fake audio it hasn’t seen before. Tests showed this method lowers detection errors compared to earlier approaches.
What this means in practice
- •For cybersecurity teams: Detect previously unseen fake speech attacks to protect voice authentication systems from fraud.
- •For telecommunication providers: Integrate advanced detection to prevent deepfake voice fraud in customer service and call centers.
Authors
Minh-Xuan Phan, Khalid Zaman, Candy Olivia Mawalim, Masashi Unoki
Abstract
Recent advances in speech synthesis and voice conversion have made deepfake speech increasingly realistic, making generalization to unseen spoofing attacks a critical challenge. Pretrained speech and audio models offer a promising direction for improving robustness to such unseen attacks. Self-supervised learning (SSL) models capture fine-grained, low-level acoustic characteristics, whereas Auditory Large Language Models (ALLMs) provide higher-level contextual representations. These complementary views can provide useful cues for improving generalization to unseen attacks. However, direct fusion does not explicitly disentangle information shared across the two views from view-specific complementary information, limiting effective cross-view integration. To address this, we propose CRAF, a cross-view residual-aware fusion framework that uses ALLM-guided cross-view attention to enrich SSL representations and adopts ALLM as a high-level reference to separate ALLM-explainable information from complementary SSL residual information. The residual is selectively refined through adaptive gating and integrated through SSL-primary fusion. Experiments on ASVspoof 5 show that CRAF with Kimi-Audio achieves an EER of 5.96% and a minDCF of 0.1192, demonstrating robustness to unseen spoofing attacks.