Speech replay attack detection adapts to changing environments over time

Domain-Incremental Learning for Multi-Channel Replay Speech Detection

Cryptography and SecuritySound

Summary

Replay attacks trick voice-controlled systems by playing back recorded speech, and detecting these attacks is harder because different rooms and places change how the sound is heard. The authors looked at how to make a detector learn to recognize these attacks in many different acoustic settings one after another without needing to keep all old recordings. They found that simply retraining forgets old environments quite badly, but using special methods that keep separate sound filters for each environment helps the detector remember better and perform more accurately. Their work tested this idea across many orders of environments, showing that the last place learned has a big impact on how well the detector works.

What this means in practice

  • For speech security engineers: Update replay speech detectors to adapt over time to different acoustic environments without storing all past data, improving long-term reliability.
  • For audio device manufacturers: Design multi-microphone devices with environment-specific acoustic processing to enhance replay attack detection performance across diverse settings.

Authors

Michael Neri

Abstract

Replay attacks are the most accessible threat to voice-controlled systems, and the acoustic cues that expose them are strongly modulated by the environment in which the attack is mounted. A detector deployed in the field therefore has to absorb new acoustic conditions over time, ideally without revisiting past recordings, since retaining speech indefinitely is both expensive and legally constrained. We frame this as Domain-Incremental Learning (DIL) over acoustic environments and present the first continual learning benchmark for multi-channel replay speech detection, evaluating a state-of-the-art beamformer-based detector over all 24 environment orderings of the ReMASC corpus with five seeds. Sequential fine-tuning forgets severely, raising the error rate on previously learned environments by 18.8 points. Elastic weight consolidation (EWC) halves forgetting but loses plasticity, gradient projection memory (GPM) is statistically indistinguishable from naive fine-tuning, and the proposed task-specific beamformer (TSB) that keeps one spatial front-end per environment significantly improves final and incremental accuracy. We further show that the last environment of the sequence dominates final performance. Code, results, and analysis are available at https://github.com/michaelneri/replay-speech-continual.
View PDFOpen arXiv