Speech model detects which words are spoofed without harming transcription
ASR-Protected Fusion and Reference-Anchored Policy Optimization for Joint ASR and Word-Level Speech Anti-Spoofing
Sound
Summary
Sometimes, fake or altered speech can trick systems that listen and transcribe what people say. Existing methods either only say if a whole sentence is fake or guess when parts are fake without matching exact words, which can lead to mistakes. The authors created a new model that listens to speech, writes out the words, and marks exactly which words are fake all at once. They use a special technique to add fake-speech clues without messing up the transcription, and a training method that helps the model get better at spotting fakes while still transcribing well. Tests show this approach works across many datasets.
What this means in practice
- •For voice assistant teams: Identify exactly which words in spoken commands are fake to avoid acting on altered speech while maintaining accurate transcription.
- •For biometric security engineers: Enhance voice authentication systems by detecting word-level speech spoofs directly linked to transcripts to improve fraud detection.$Commercial implications: Enables development of more secure voice authentication products by pinpointing spoofed words during authentication conversations.
Authors
Xueping Zhang, Xiaoxiao Miao, Tiantian Feng, Ming Li
Abstract
Existing anti-spoofing systems report either an utterance-level score or a time-based spoof interval: the former cannot show which words are spoofed, while the latter does not directly match the transcript and must be post-aligned with words, so alignment errors can lead to wrong word-level labels. Standard ASR model outputs only text; spoof information in the audio is lost after recognition and is not passed with the transcript. Yet changing only a few words can change the meaning of a sentence, leaving downstream systems to use incorrect text without knowing which words are spoofed. We therefore treat partial anti-spoofing as tagged ASR: the model transcribes speech and marks spoofed words at the same time, directly linking spoof labels to output words. We adopt the audio LLM and use ASR-Protected Subspace Gated Fusion (APSGF) to fuse spoof cues with ASR features. Spoof cues are added only outside an ASR-protected subspace, so they do not disturb recognition features. After cross-entropy training, we fine-tune the model with Reference-Anchored Policy Optimization (RAPO). Since the SFT policy is already concentrated, multiple samples from the same input tend to receive similar rewards, resulting in weak group-wise advantages. RAPO instead generates a single sample with low sampling randomness and anchors its advantage to the reward of a reference model. This allows the model to reinforce responses that outperform the reference while suppressing those that underperform it. Experiments on five datasets demonstrate the effectiveness of our approach in jointly detecting spoofed words and preserving ASR accuracy. The code is publicly available.