Video models match human accuracy but use different thinking methods
Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning
Computer Vision and Pattern Recognition
Summary
Video AI models can now predict physical events in videos almost as accurately as humans, but they do not think like people do. The authors found that while models make predictions well on some tasks, they struggle with others that require understanding unseen physical forces, like gravity or cause-and-effect chains. These models seem to rely more on patterns in the visible scenes than on imagining what really happens next. This shows that scoring high on accuracy doesn't mean models reason like humans.
What this means in practice
- •For ai system designers: Improve video reasoning models by targeting gaps where current architectures fail to simulate physical dynamics as humans do.
- •For video analytics developers: Develop better evaluation methods for video models that go beyond accuracy to reveal reasoning strategies and weaknesses.
Authors
Fanhong Li, Shurui Zheng, Zi Yin, Junbo Cui, Lei Ji, Jia Liu
Abstract
Video foundation models now reach human-level accuracy on physical-reasoning benchmarks, yet such tasks require predicting unobserved physical outcomes. Do these models perform human-like forward simulation, or do they exploit statistical regularities in visible scenes? Accuracy alone cannot distinguish these strategies. We introduce a distributional evaluation framework that treats model seeds and human raters as populations, enabling comparison of consensus, uncertainty, and strategy. On the Physion benchmark, we evaluate three ViT-L architectures (V-JEPA2, VideoMAEv2, DINOv2). V-JEPA2 narrows the accuracy gap to ~1 percentage point (73.2% vs. 74.2%), yet model-human disagreement reaches 26.4%, far exceeding human-human disagreement (4.8%), with substantially lower agreement (kappa ~ 0.48 vs. 0.91). The divergence follows forward-simulation demands: models outperform humans on geometric reasoning (linking, +11.8 pp) but underperform on gravitational dynamics (rolling, -11.8 pp) and causal chains (dominoes, -10.5 pp). Strategy fingerprinting confirms all three architectures share non-human strategies while none aligns with humans. Attribution analysis suggests that unobservable outcome features, rather than visible scene properties, predict this divergence, consistent with models relying more on scene-level statistical regularities than on explicit forward simulation, a systematic divergence that accuracy alone cannot reveal. Code is available at https://github.com/fanhong-li/model-human-divergence.