Visualtrust improves video question answering under misleading text claims
VisualTrust: Video Reasoning under On-Screen Claim Framing
Computer Vision and Pattern Recognition
Summary
Sometimes videos have on-screen text that can describe what is happening or try to change the meaning of the video. The authors found that many AI models blindly trust this text even when it conflicts with what is actually shown. To test this, they created VisualTrust, a dataset with videos paired with accurate or misleading text claims. They also designed DEAR, a method that checks evidence from video and text separately before answering, which helps models resist being fooled by misleading text.
What this means in practice
- •For video content moderation teams: Improve detection of misleading video captions that misrepresent visual content to protect viewer trust.
- •For automated fact-checking services: Enhance fact-checking accuracy by verifying on-screen text claims against underlying video evidence.
Authors
Huachao Zhu, Junjielong Xu, Sicheng Li, Liang Yang, Mingtao Chen, Zhichao Hu, Li Jiang, Pinjia He
Abstract
On-screen text can describe what a video shows, or it can frame what the video is supposed to mean. Current benchmarks treat both alike, testing whether a model can read the text rather than whether it should trust it. We study claim deference, multimodal large language models' tendency to let an on-screen claim override conflicting visual evidence. We introduce VisualTrust, a counterfactual benchmark with 5,063 cases across 14 claim-fact classes. Each case pairs a question with three versions of the same video: one with no claim, one with an accurate claim, and one with a misleading claim that alters a single grounded fact. The option the misleading claim supports is the trap answer. Across seven open-weight and four proprietary models, mean accuracy falls from 90.1% without a claim to 51.9% with a misleading claim. Among initially correct responses, 42.8% flip to the trap answer and only 0.6% become other errors. Strong no-claim accuracy does not ensure resistance. We also introduce DEAR (Decoupled Evidence Arbitration and Reading), a training-free agent for video question answering that separates evidence arbitration from reading. An independent check reports conflicting evidence, a selector weighs it without a fixed modality priority and blinds what it withholds, and the reader answers from the resulting video in a fresh context. On 500 videos across three proprietary models, DEAR raises accuracy under misleading claims from 47.2--64.0% to 81.8--85.2% and reduces trap-answer selection from 34.4--50.8% to 11.6--15.0%, while preserving no-claim accuracy.