What's the Catch? Evaluating Temporal Consistency in Vision-Language Models
2026-08-24 • Computation and Language
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
AI summaryⓘ
The authors studied if vision-language models (VLMs) can understand the order and timing of events in videos. They made a test called TimeCatch where videos have out-of-place or noisy frames to see if the models notice these mistakes. The authors found that VLMs can spot weird frames but have trouble realizing when frames are in the wrong order, unlike humans who do this well. Their results suggest current VLMs don't fully grasp how things change over time in videos. TimeCatch offers a new way to measure how well these models understand temporal consistency.
vision-language modelstemporal groundinganomaly detectiontemporal consistencyframe-level anomalyvideo sequencelocalizationsynthetic datasetsGaussian noisehuman study
Authors
Marek Hradil, Danae Sánchez Villegas
Abstract
Vision-language models (VLMs) achieve strong performance on video and image-sequence benchmarks, yet it remains unclear whether they capture temporal structure. To study this question, we formulate temporal grounding as an anomaly detection problem, providing a simple and controlled evaluation that directly tests sensitivity to temporal consistency. We introduce TimeCatch, where temporal anomalies are created by swapping consecutive frames and frame-level anomalies by replacing a frame with Gaussian noise. Models are evaluated on anomaly detection and localization tasks across four synthetic and real-world datasets, alongside a human study. Our evaluation reveals a substantial gap between frame-level and temporal anomaly detection. While VLMs consistently detect frame-level anomalies and often localize them accurately, they perform near chance on temporal anomaly detection and only modestly above chance on localization. Humans, in contrast, achieve near-ceiling performance on both tasks. Additional analyses across model scales, prompting strategies, sequence lengths, and visual similarity suggest that these failures cannot be explained solely by limitations in perception or model capacity. Together, these findings indicate that current VLMs can identify anomalies within individual frames but struggle to integrate information across frames to reason about temporal consistency. TimeCatch provides a controlled benchmark for evaluating temporal grounding in vision-language models.