Video world models struggle to remember multiple objects over time
OPIS: An Input-Grounded Benchmark for Multi-Object Memory in Video World Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Remembering where many specific objects are in a video is hard for computer models. The authors created OPIS, a new way to test how well models keep track of individual objects from the start of a video. They tested models on 500 videos with many objects across real, robot, and game scenes, measuring if the models keep track of object presence, identity, and structure. Results show that as the number of objects grows, it gets much harder for models to remember the right details instead of just making something that looks plausible.
What this means in practice
- •For robotics engineers: Use the OPIS benchmark to evaluate and improve robot perception systems that need to remember multiple objects over time in complex environments.
- •For game developers: Assess game AI systems’ ability to track objects accurately for enhanced interactive environments and more consistent game worlds.
Authors
Hao Wang, Tao Yu, Liuzhou Zhang, HeXin Wang, Haopeng Jin, Yuxuan Zhou, Xinming Wang, Hongzhu Yi, Xinye Li, Yuanlei Wang, Ping Nie, Yan Huang, Yuxuan Zhang, Pengfei Zhou, Yanyan Zou, Wei Yang
Abstract
Video world models must preserve the visual state of the world over time, but existing evaluation protocols often rely on generated histories, video reference, or selected revisit viewpoints that can confound the assessment of a model's true memory capability. To address this, we introduce OPIS, an input-grounded benchmark that strictly anchors the assessment to a fixed set of object instances from the initial observation for evaluating multi-object memory in video world models. The OPIS dataset comprises 500 cases across real-world, embodied-robotic, and game-world domains, providing dense object-level annotations for 12,672 rigid, articulated, and deformable instances. Our object-centric evaluator combines association and explicit visibility reasoning to hierarchically measure Object (O) Presence (P), Identity (I), and Structure (S), utilizing static or dynamic evaluation tracks based on object kinematics. Across eight image-to-video or camera-conditioned world models, our proposed OPIS scores range from 48.65 to 56.01. As the reference inventory grows from less than 20 to more than 40 objects, the Presence, Identity, and Structure scores show an overall decline, with the average Identity score falling from 40.22 to 23.11. The results demonstrate that preserving the particular object instances in the input is considerably harder than generating plausible visual elements.