Speech fact checking improves with reasoning on retrieved evidence
To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech
Machine LearningArtificial IntelligenceComputation and LanguageSound
Summary
Misinformation spreads not only in writing but also in spoken content like podcasts and speeches. The authors created VeriSpeak, a test collection of spoken statements, to see if AI models that fact-check text can also do the same with speech. They found that these models struggle more when checking spoken claims directly, especially when just given related text information to help. However, combining this text retrieval with clear step-by-step reasoning helps models verify speech-based claims better.
What this means in practice
- •For content moderation teams: Use VeriSpeak-trained models to check spoken claims in podcasts and videos for misinformation.
- •For voice assistant developers: Improve virtual assistants to verify information heard in voice queries by combining speech input with retrieved text evidence and reasoning.
Authors
Debajyoti Mazumder, Mamta, Abhirama Subramanyam Penamakuri
Abstract
Online misinformation increasingly appears in spoken formats such as news clips, podcasts, interviews, political speeches, and social media videos, creating a need for fact-checking systems that can verify claims directly from speech. We introduce VeriSpeak, a probe benchmark for studying speech-based fact verification in Large Audio Language Models (LALMs). VeriSpeak contains 3,879 spoken claims spanning temporal, geographical, and relational facts, with balanced true and false labels. The benchmark is designed to examine whether factual verification ability transfers from text to speech, and whether retrieval-augmented LALMs can use textual evidence to correctly support or refute spoken claims. Our experiments reveal a consistent text-speech modality gap: LALMs that verify written claims reliably often fail on the same claims when spoken. Moreover, retrieval alone provides limited gains because models frequently conflate retrieved evidence with the spoken claim. In contrast, retrieval combined with explicit reasoning improves claim-evidence comparison, with a thinking-tuned LALM reaching 86.1% accuracy. VeriSpeak highlights that effective speech misinformation detection requires not only speech understanding, but also grounded reasoning over retrieved evidence. The dataset is publicly available via Hugging Face at https://huggingface.co/datasets/abhiram4572/VeriSpeak.