Video large language models struggle to understand motion in clips
MotionBlind: Probing the Illusion of Motion Understanding in Video-LLMs
Computer Vision and Pattern Recognition
Summary
Video large language models (Video-LLMs) are tools designed to interpret videos, including how things move within them. This paper shows that most Video-LLMs can't reliably tell differences in motion like speed or direction between very similar clips. The authors created a new test called MotionBlind that specifically checks if these models understand motion by comparing pairs of nearly identical videos that differ only in how things move. They found that most models performed no better than guessing, except one newer model that still had trouble with speed. This means current Video-LLMs are not yet reliable at reasoning about motion in videos.
What this means in practice
- •For video system developers: Validate video processing systems' ability to detect motion accurately using the MotionBlind benchmark to improve perceptual models.
- •For robotics engineers: Evaluate and select video perception models that genuinely understand motion for better control and planning in robots.
Authors
Dhairya Bhatia, Bishoy Galoaa, Oliver Fritsche, Shahid Kamal, Muhammad Obaidullah Abdul Salam, Umer Saleem, Om Rastogi, Frania Felix Chettiar, Nesli Erdogmus, Sarah Ostadabbas
Abstract
Video large language models (Video-LLMs) are increasingly used as the perceptual front end of world models, a role that assumes they can read motion: how fast something moves, which way it travels, how hard it is pushed. We show they cannot. A Video-LLM can watch two clips of the same person in the same room, name every object in both, and still fail to say which clip moves faster. We introduce MotionBlind, a contrastive benchmark of self-recorded video for physically grounded motion(speed, magnitude, and direction), the variables a world model must predict. Each instance is a pair of near-identical clips that differ only in motion. Each clip carries two complementary yes/no questions, giving four items per instance, and a model earns credit only if all four are correct. We report Instance Accuracy(IAcc), which has a 6.25% chance floor. Single-frame, appearance, and language-only shortcuts all collapse to it. MotionBlind complements the recent TimeBlind benchmark. We run a controlled study of six open and two frontier Video-LLMs, varying whether the video is present, whether frames are shown in the correct temporal order, and how frames are sampled (1 to 24 frames, four selection strategies). Open models sit near the 6.25% floor, and scale does not help. Removing the video drops every model to zero IAcc, and shuffling frames collapses IAcc to chance, so the task genuinely needs video in order. Neither more frames nor smarter frame selection closes the gap, because these change which frames are seen, not whether motion is read. Only Gemini3.1 Pro clears the benchmark overall, and even it fails on speed. A frontend that cannot tell two speeds of the same action apart is not yet a trustworthy source of supervision, reward, or evaluation for a world model.