4DMemory benchmark measures AI memory for long videos
4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and LanguageMultimedia
Summary
Current AI models find it hard to remember details over very long videos or events that last hours. The authors created a big new dataset called LongVideo with 900 hours of video and thousands of questions to test how well AI keeps track of time and space details. They also built 4DMemory Harness, a tool that combines many specialized AI models to help with long-term reasoning. When tested, most large AI models alone struggled, but the authors’ tool helped improve their performance a lot. This work aims to push AI toward understanding and reasoning about long-term video events better.
What this means in practice
- •For video analytics developers: Test AI systems’ ability to understand and remember long video content for improved video analysis tools.
- •For robotic system engineers: Build agents that can recall long sequences of events and predict future states in complex environments.
Authors
Yanhao Jia, Yi Zhu, Yiran Zhong, Weigao Sun, Feida Zhu, Pengkun Jiao, Sihan Yang, Ming Ma, Steven Hoi, Erik Cambria
Abstract
Humans efficiently process multimodal streams spanning hours to years, building persistent, fine-grained spatiotemporal memories that support reasoning about and predicting long-horizon events. In contrast, existing benchmarks focus on short- and medium-context tasks, and most foundation models struggle to preserve fine-grained information over long horizons. To evaluate and improve long-horizon spatiotemporal reasoning, we introduce the LongVideo dataset and the 4DMemoryBench. LongVideo comprises about 900 hours of video and over 5,000 manually reviewed questions covering visual, textual, and spatiotemporal reasoning across diverse context lengths. We further develop the 4DMemory Harness, which integrates four complementary skills, 22+ specialized tool models, and a model-specific retrieval-augmented generation system. Evaluating 13 leading models, we find that standalone foundation models struggle with long-horizon tasks, while the 4DMemory Harness improves all of them by over 50\%. Compared to previous benchmark, the same models exhibit a performance drop of more than 10\% on 4DMemoryBench. We hope 4DMemory will facilitate the development of spatiotemporal agents capable of addressing complex, long-horizon tasks.