Video memory can aid models with generic or exact content retrieval
Does Video Memory Use What It Retrieves? A Causal Audit of Memory Specificity
Computer Vision and Pattern Recognition
Summary
Video models use memory to remember what happened earlier in long video clips. This paper checks if the improvement from memory comes from the model actually using the correct past details, or just from having some memory at all. The authors swap out the memory used during testing to see how important the exact stored content is. They find that sometimes the benefit comes from replacing or fixing general representations, sometimes from related context, and sometimes from the exact past scenes. This helps understand how video models really use memory.
What this means in practice
- •For machine learning engineers: Improve video prediction systems by choosing memory strategies targeted at representation repair or exact content retrieval depending on application needs.
- •For computer vision developers: Design video segmentation tools with better memory modules that balance broad memory context and precise episodic content for improved accuracy.
Authors
Aditi Tiwari, Akshit Bhalla, Darshan Prasad, Heng Ji
Abstract
Video models increasingly use memory to preserve information over long sequences, with the assumption that gains come from retrieving and using the correct past content. Standard memory ablations test whether memory helps, but not whether the retrieved content is responsible. We test this directly with read-time memory substitution, which replaces the consumed memory value while leaving the rest of the computation unchanged. This separates memory benefit from memory specificity, the extent to which the gain depends on retrieved content. Across frozen video world models, identity-free controls containing no evaluation-specific content recover essentially the full benefit on Ego-Exo4D and 7-Scenes and about 70% on TUM. In the Ego-Exo4D dose response, recovery falls from 102% to 1% as these values move away from observed training-memory representations, supporting representation repair as the best-supported explanation in this setting. WorldMem shows graded dependence. A wrong memory from the same trajectory recovers 94.1% of the PSNR benefit relative to zero content, while a donor from a disjoint trajectory and biome recovers 43.7%. SAM 2 shows strong content dependence. On DAVIS, replacing the correct spatial memory with a valid wrong memory reduces mean region and boundary score from 0.926 to 0.182. At MOSEv2 reappearance, it falls from 0.459 to 0.000. These results show that memory gains can depend on generic representation support, broader context, or exact episodic content. Read-time substitution provides a direct way to distinguish them.