Video language models struggle to understand motion in driving tasks
Mind the Temporal Gap: Evaluating Video Models for Temporal Understanding in Driving
Computer Vision and Pattern RecognitionRobotics
Summary
Video-language AI models are meant to understand both what is happening in a video and the timing of actions. The authors found that many current models have trouble reliably recognizing motion and temporal events in driving videos, often relying on single frames instead of changes over time. However, when the models are slightly fine-tuned or given extra training on specific features, they can better detect motion patterns. This means that although the models have the ability to recognize motion cues, their usual ways of answering questions about driving videos are still unreliable.
What this means in practice
- •For automotive safety teams: Assess and improve video-based driver assistance systems by testing temporal motion understanding with targeted fine-tuning techniques.
- •For video ai developers: Design video-language models that incorporate intermediate visual feature probes to better capture motion cues in safety-critical contexts like driving.
Authors
Shiva Sreeram, Erfan Aasi, Tsun-Hsuan Wang, Alaa Maalouf, Daniela Rus
Abstract
Modern video-language models are increasingly treated as general-purpose video reasoners, yet it remains unclear whether they robustly capture temporal dynamics in safety-critical settings such as driving. We study this question through controlled ego-motion tasks that require identifying actions from driving videos. Across video-language models, we find that standard prompting, visual prompting, few-shot prompting, and chain-of-thought prompting remain unreliable and highly task- and model-dependent, with persistent class biases, although newer models show substantial task-specific few-shot gains. In contrast, lightweight LoRA finetuning and simple probes trained on intermediate visual features recover strong motion-discriminative performance, indicating that motion-relevant cues are often present inside the models but are not reliably used by the final video-to-language pathway. These results suggest that, in driving, current video-language models remain unreliable temporal reasoners and often behave more like frame-level, bias-driven recognizers than dependable temporal world models.