Temporal coherence analysis distinguishes real and fake audio reliably
Audio Deepfake Detection Using Temporal Coherence Analysis
Cryptography and SecuritySound
Summary
Fake audio created by AI can trick people and cause problems like fraud or copyright issues. This paper presents a way to spot fake audio by looking at how parts of a sound clip relate to each other over time, using patterns learned from both speech and music. The method uses simple math tools and clever statistics to tell real and fake sounds apart. It also finds surprising behavior where some features flip their meaning when tested in the real world, and that speech and music have opposite clues for detecting fakes.
What this means in practice
- •For audio security teams: Detect synthetic voices or music in security systems by analyzing temporal relationships in audio segments.
- •For music copyright analysts: Identify AI-generated music tracks to protect intellectual property by leveraging temporal coherence features across sound segments.
Authors
Justin D. Norman, Sarah Barrington
Abstract
The proliferation of AI-generated audio (so-called "deepfake" audio) poses significant threats to information integrity, from voice cloning fraud to synthetic music copyright disputes. We present a temporal coherence analysis framework built upon Contrastive Language-Audio Pretraining (CLAP) embeddings that spans speech, instrumental music, and music with vocals. By computing pairwise cosine similarities between audio segment embeddings and extracting statistical features from the resulting distributions, we train lightweight ensemble classifiers that reliably distinguish authentic from synthetic audio. Our work provides an interpretable, computationally efficient alternative to common deep learning methods while still achieving competitive performance across speech and music domains. Further, we reveal two notable empirical findings about audio deepfakes: (1) a feature-label inversion phenomenon in which 21 of 29 statistical features reverse their discriminative direction between training and in-the-wild deployment, and (2) a speech--music direction reversal in which entropy discriminates in opposite directions for speech and music deepfakes.