Summary
Video models are supposed to understand things that change over time, like how objects move in a video. However, sometimes what looks like understanding motion is actually just detecting different positions without truly capturing the movement. The authors came up with a new statistical test called No-False-Positives (NFP) to check if features in these models really represent temporal concepts. They applied this test to several video models and confirmed that they do encode true temporal features, while an image-only model did not. The findings help understand how video models process time-related information, though using these features to improve performance showed mixed results.
What this means in practice
- •For video model developers: Use the NFP test to verify video models truly capture temporal motion features rather than mere spatial correlations.
- •For computer vision engineers: Assess and debug features in video analysis pipelines to confirm they encode actual temporal dynamics of objects.
Abstract
Video models have the architecture to support encoding temporal aspects of the input, i.e. concepts that require more than one frame to define, like object movement. However, because motion and appearance change are entangled, a feature that appears to encode temporal concepts like object movement may instead encode non-temporal concepts like spatial presence at different positions whose activations merely correlate with motion. To resolve this, we introduce the No-False-Positives (NFP) test, a statistical test with a formal guarantee of statistical specificity for identifying features encoding temporal concepts. Using specially designed stimuli, we apply this test to representations produced by VideoMAE, V-JEPA2, and TimeSformer, and observe that temporal features exist in each model. We validate the test first by showing it yields no temporal features for the DINOv2 image-encoder. Then, using a synthetic representation space, we show that detection rate increases with a direction's projection onto the temporal subspace, and that Sparse Autoencoder features test-flagged as temporal project much more of their norm onto the temporal subspace than unflagged features. Interventions on temporal SAE features yield mixed results across models, suggesting that intervening on temporal features may not outperform a non-temporal feature baseline.