System detects fake audio snippets in multi-speaker talks quickly
Zero-Shot Temporal Localisation of Audio Deepfakes in Multi-Speaker Conversations
SoundMachine Learning
Summary
Voice cloning fraud is becoming more subtle by only replacing small parts of real conversations with fake speech. The authors address the challenge of finding exactly where these fake segments appear in multi-person recordings. They created a method that uses an existing detector without retraining and identifies the fake parts in conversations with high accuracy. Their approach works well across different detectors and datasets, establishing a benchmark for this problem.
What this means in practice
- •For audio security teams: Detect precisely where fake audio segments occur in multi-speaker recorded conversations to improve fraud investigation.
- •For call center supervisors: Monitor customer interactions to pinpoint inserted synthetic speech without needing new training data, enhancing trust in recordings.
Authors
Soumyadeep Roy
Abstract
Voice-cloning fraud increasingly relies on surgical injection: a genuine conversation in which only one or two sentences are replaced by synthetic speech. Utterance-level deepfake detectors emit a single real/fake label per clip and cannot report where the synthetic speech lies. We formalise this as Temporal Deepfake Localisation in Multi-Speaker Conversations (TDLMC), show that equal error rate and min-DCF are ill-posed once a file contains both classes, and propose temporal metrics for this regime. Our contribution is a training-free five-stage pipeline that wraps a frozen binary detector and adds segment-level output with no retraining, using a two-threshold hysteresis finitestate-machine decoder to turn noisy window scores into coherent intervals. On 180 constructed multi-speaker conversations from ASVspoof 5, the system attains temporal intersection-over-union 0.90, temporal detection rate 0.95, and MS-DCF 0.26 with a strong backbone, and its false-alarm rate on genuine speech is below 6%, falling under 2% on genuine real multi-speaker dialogue (AMI). Under an identical pipeline, a trained localiser improves temporal IoU by only about 0.04, bounding the cost of forgoing supervision. Evaluated across three frozen detectors under one decoder whose constants are selected on a held-out calibration split, and with a controlled analysis attributing the residual false-alarm rate to a backbone domain gap rather than to the decoder, this provides the first zero-shot baseline and a reusable benchmark for TDLMC.