Large diverse dataset and interactive world model improve embodied agents
EmbodiedWorld-200K: Toward Embodied World Agents via Embodied-Planning Dataset and Interactive World Models
Artificial Intelligence
Summary
Teaching robots to understand and act in many different 3D environments is hard because most studies focus on one scene or one task. The authors created a huge dataset with over 200,000 examples from many different scenes, helping robots learn from both simple motions and complex instructions. They also developed a way for robots to keep learning by interacting with a video-based simulation instead of expensive real-world setups. This combined training helps robots plan and act better, especially in new or unfamiliar places.
What this means in practice
- •For robotics teams: Train robots to navigate and plan actions in varied real-world environments using a large annotated dataset and interactive simulated worlds.
- •For game developers: Use video world models to cheaply test and improve AI agents' interactive behaviors across multiple virtual scenes.
Authors
X. Feng, J. Tang, Z. Lin, L. Chen, S. Shao, Q. Lu, K. Huang
Abstract
Building embodied agents that can perceive, reason, and act across diverse 3D worlds is a central research direction toward artificial general intelligence. Yet most existing work remains confined to specific tasks in a single scene and transfers poorly to new environments, leaving general embodied agents for open-world settings under-explored. We refer to such agents as Embodied World Agents (EWAs), whose defining property is the ability to understand diverse visual environments and execute general actions in open-world conditions. To advance EWA research, we study open-world embodied navigation-planning and release EmbodiedWorld-200K, a large-scale dataset of over 200K samples spanning diverse scenes. Each example is equipped with multi-level annotations, ranging from low-level camera-motion trajectories to high-level semantic instructions. The dataset also provides unified evaluation splits and task-specific metrics, supporting offline pretraining and fair evaluation of EWAs. Beyond offline supervision, a defining characteristic of an EWA is its interaction with the environment; training solely on offline data is therefore insufficient, especially for generalizing to scenes not covered in the training distribution. Building dedicated real interactive environments, however, is often prohibitively expensive. To address this, we propose a novel training scheme that employs a video world model as a low-cost environment proxy and performs interactive reinforcement-learning (RL) post-training of the EWA within it. Empirically, pretraining on EmbodiedWorld-200K substantially improves the open-world embodied planning ability of the EWA, and further RL post-training inside a world model yields clear gains on out-of-distribution scenes. The dataset, baseline checkpoints, and training and evaluation code are available at https://xiaokunfeng.github.io/EmbodiedWorld-200K/.