Video models struggle to explain magic tricks with misdirection
The Prestige: Benchmarking Cognitive Visual Reasoning using Magic Tricks
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Tracking what happens in videos can be hard when things are meant to be tricky, like magic tricks that try to confuse. The Prestige benchmark tests video understanding models by showing them magic performances and asking questions about what really happened and why. The researchers found that even the best models have a long way to go before they understand these tricky videos as well as humans do. They also noticed that models sometimes pick the right answers without explaining them properly, showing that how we ask questions matters for evaluating video understanding.
What this means in practice
- •For video analysis teams: Improve video understanding tools by testing models on complex events involving misdirection and causality found in magic performances.
- •For ai evaluation developers: Develop more reliable and rigorous benchmarks for cognitive visual reasoning in video understanding systems.
Authors
Shuo Wen, Beixi Du, Edwin Meriaux, Junming Shi, Chloe Si, David Meger, Doina Precup, Gregory Dudek
Abstract
We introduce The Prestige, a video understanding benchmark that probes cognitive visual reasoning through professional magic performances, where deliberate misdirection challenges models to track events and explain unexpected outcomes. The benchmark comprises open-ended questions ranging from temporal and causal understanding to robustness against adversarial questions. Evaluation of leading open-source and proprietary VLMs reveals a substantial gap between even the strongest model and human performance. We also find systematic discrepancies between human and automated grading and identify recurring model failure modes. Ablations further show that models can select correct multiple-choice answers even when they fail to generate correct explanations independently, motivating further investigations into question formats. Together, these findings highlight limitations in both VLM reasoning and its evaluation, underscoring the need for more reliable video understanding models and more rigorous benchmarks.