Human experience drives learning world structure and changes
Holi-World: Building Holistic World Intelligence from Human-Ego Experience
Computer Vision and Pattern Recognition
Summary
Understanding how the world and objects change over time is hard because it requires lots of complex information and annotations. The authors created HOLI-WORLD, a system that collects and organizes recordings of human experiences to automatically build detailed records of scenes and object interactions. They used this data to train AI models that better understand spatial and temporal changes in dynamic environments. Their results show improved performance in tasks that involve recognizing static scenes and tracking changes over time.
What this means in practice
- •For robotics engineers: Develop robots that better understand changing environments by using AI trained on rich human experience data for improved object tracking and interaction.
- •For augmented reality developers: Create AR applications that more accurately map and interact with dynamic real-world scenes based on detailed world records from human egocentric data.
Authors
Hao Li, Xiyuan Zheng, Yuanyuan Gao, Yalun Dai, Xinhao Ji, Xuan-Li Chen, Jing Zhao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu
Abstract
Human egocentric experience offers a rich foundation for learning how the world is organized and how it changes through interaction. Realizing this potential at scale requires turning growing volumes of recordings into learning resources that connect scene structure, object semantics, and interaction histories across viewpoints and time, while reducing dependence on costly manual annotation. We introduce HOLI-WORLD, an integrated ecosystem connecting experience capture, world representation, automated data construction, model learning, and independent evaluation. To sup- port learning at scale, HOLIWORLD-Foundry combines complementary static and dynamic perception to automatically construct world records that connect persistent scene structure and evolving object states with interaction histories and supporting observations. HOLIWORLD-Data organizes these records from approximately 100 self-collected scenes into grounded learning resources, including approximately 80K QA pairs used to train HOLIWORLD-VL-4B. Independent human annotations of heldout recordings form HOLIWORLD-Bench, supporting evaluation of both annotation quality and learned spatial and temporal understanding. Experiments on ScanNet, ScanNet++, Aria Digital Twin, and HOLIWORLD-Bench demonstrate strong performance in static perception and dynamic tracking, while training on HOLIWORLD-Data substantially improves HOLIWORLD-VL-4B's spatial and temporal reasoning on HOLI- WORLD-Bench and EgoDynamic4D. HOLIWORLD provides a practical foundation for expanding learning resources from the continuous flow of human experience and advancing machine understanding of the dynamic world.