Video models help understand and change the world through video
Reading, Writing, Sharing, and Interacting with the World through Video
Computer Vision and Pattern Recognition
Summary
Videos show us parts of the world over time, but they don't capture everything perfectly. The authors look at research about how computers can read videos to learn about the world, write videos based on what they know, share knowledge between reading and writing, and interact by predicting changes when actions happen. They emphasize the difference between making videos look smooth and making sure the world they represent stays consistent and reasonable. Their review points to new challenges like producing videos that follow physical rules and are useful for decisions.
What this means in practice
- •For video game developers: Create game worlds where video scenes can be controlled and updated according to player actions using models that understand and generate video dynamically.
- •For robotics engineers: Build robots that interpret camera input as partial views of the environment and predict future states to plan interactions more effectively.
A survey. It maps existing work.
Authors
Shengqiong Wu, Meng Luo, Bobo Li, Ziwei Liu, Shuicheng Yan, Ming-Hsuan Yang, Philip Torr, Hao Fei
Abstract
Video foundation models have advanced rapidly in understanding and generating visual sequences, while video world models seek to capture the underlying dynamics of the world they depict. Despite their different objectives, both concern the relationship between video observations and an evolving world. This survey adopts world modeling as a unifying lens for examining these developments. Viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, we organize the literature around four complementary operations: reading infers world properties from observed video; writing synthesizes video from specified or learned world conditions; sharing integrates understanding and generation through common modeling mechanisms; and interacting predicts and updates world states under actions and evolving observations. This perspective distinguishes temporal coherence in observed or generated sequences from consistency in the underlying world state. Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow.