Textual Acoustic Grounding for Generalizable LLM-Based Deepfake Voice Detection
2026-08-31 • Sound
Sound
AI summaryⓘ
The authors studied how to better detect fake voices, especially when testing on new, unseen types of audio. They found that simply tuning large language models (LLMs) on audio can lead to overfitting, so keeping the LLM fixed works better and uses less resources. To improve understanding between audio sounds and language models, they added special acoustic features as text tokens, which helped the models perform much better. Their method achieved much stronger results on tough tests while still doing well on familiar data.
deepfake voice detectionaudio large language modelsmodalitiesfine-tuningcross-modal promptingopenSMILEmacro-F1 scoreout-of-domain generalizationQwen LLMsacoustic features
Authors
Yassine El Kheir, Xin Wang, Wanqing Ge, Tim Polzehl, Sebastian Moeller, Junichi Yamagishi
Abstract
Deepfake voice detection suffers from poor generalization across unseen domains. While Audio Large Language Models (ALLMs) show promise, the modality gap between continuous audio embeddings which capture the subtle acoustic details necessary for deepfake detection and the semantic space of LLMs remains a critical, underexplored bottleneck. We address this by benchmarking diverse audio encoders integrated with Qwen LLMs (0.5B to 7B parameters). First, we demonstrate that fine-tuning the LLM alone risks out-of-domain overfitting, making a frozen LLM a stronger, resource-efficient baseline. Second, to explicitly bridge the modality gap, we introduce a cross-modal prompting strategy that injects linguistic-knowledge-driven acoustic features (via openSMILE) as structured text tokens. This explicit textual grounding not only enhances the frozen baseline but also makes LLM fine-tuning more effective. Ultimately, our approach demonstrates state-of-the-art resilience on the out-of-domain ITW and MLAAD benchmarks, yielding over \textbf{16.2\%} absolute improvement in Macro-F1 over existing ALLM baselines while maintaining competitive in-domain performance. All models reported in this work are \href{https://huggingface.co/01Yassine/AudioLLM-Deepfake-Detection}{publicly available}.