SceneForesee system predicts how scenes change over time
SceneForesee: Learning Scene Evolution for Spatiotemporal Reasoning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding how things move and interact in a video over time is a difficult problem. The authors created a new task called Scene Evolution Understanding (SEU) to predict what happens next in a scene, including the paths of objects and the events they undergo. They built a large dataset called SceneForesee with many videos and labeled object identities, trajectories, and events to test this task. They also designed a system, EvoSee, that learns to predict future changes and improves on existing models. This work helps computers better anticipate and explain how scenes in videos evolve.
What this means in practice
- •For video analysis teams: Enhance video understanding by predicting and explaining object movements and scene changes in future frames.
- •For autonomous vehicle developers: Improve autonomous navigation by forecasting dynamic scene changes and interactions among vehicles and pedestrians.
Authors
Chaoyue Li, Shoudong Han, Haoyang Wu, Boxue Yang, Sijia Chen, Yufan Chen, Ruping Zhang
Abstract
Large multimodal models (LMMs) need to understand how entities move and interact as a scene develops. This requires following the same entities over time and relating their observed motion to subsequent changes. We formulate Scene Evolution Understanding (SEU), a task for predicting and interpreting this change. Given observed video, a model estimates entity histories and predicts one scene continuation through future trajectories and Events, accompanied by Visual Evidence from observed motion and a scene description. We construct SceneForesee, a benchmark for evaluating entity continuity and scene evolution reasoning through paired observed and future scenes with shared entity identities. It contains 144.57K temporal pairs, 1.05M entity correspondences, and 272.49K Events paired with observed evidence. We propose a spatiotemporal reasoning framework, EvoSee, that learns future geometry and Event semantics through shared candidate states and joint assignment. It selects one continuation per entity from observation and uses the selected changes for entity refinement, retention, and scene representation. On SceneForesee, EvoSee outperforms Qwen3.5-9B by 10.45 points in Combined Point-F1 and 17.24 points in Event Global F1. On the 4D reasoning benchmark Dyn-Bench, it improves question answering accuracy and grounding J&F by 4.8 and 3.58 points over their respective base systems. These results support scene evolution as both a prediction target and a dynamic representation that brings predicted entity changes into spatiotemporal reasoning.The project is available at https://github.com/mikubaka88/SceneForesee-Spatiotemporal-Reasoning.