EnvDreamer generates large virtual environments for AI training
EnvDreamer: Large-Scale Multimodal-to-Environment Generation for Embodied AI
Artificial Intelligence
Summary
Training robots and AI systems to navigate and interact with the real world is challenging due to the cost and complexity of real environments. The authors created EnvDreamer, a tool that uses advanced language and vision models to automatically build detailed virtual worlds for AI training. These environments can be diverse and interactive, allowing AI to learn tasks like navigation and object manipulation without human supervision. They also provide a large dataset of these environments to help standardize testing and benchmarking.
What this means in practice
- •For robotics development teams: Train and evaluate robot navigation and manipulation policies using scalable, diverse simulated environments generated automatically.
- •For game development studios: Generate large-scale, richly detailed interactive scenes automatically for testing game AI behavior and interaction models.
Authors
Kabir Swain, Sijie Han, Antonio Torralba
Abstract
Large datasets and high capacity models have accelerated progress in vision and language. This work introduces a platform aimed at bringing comparable gains to embodied learning, world models, and robotics. We present EnvDreamer, a framework that uses large language and vision language models to generate Unreal Engine 5 environments for embodied AI and robot training. EnvDreamer enables sampling of large, diverse, interactive, customizable, and validator passed virtual environments for training and evaluation across navigation, interaction, and manipulation. We illustrate the platform with a large set of generated scenes and simple baselines. Policies trained on EnvDreamer generated environments, without explicit mapping or human task supervision, achieve competitive results on multiple embodied benchmarks spanning navigation, rearrangement, and manipulation. EnvDreamer also supports image-conditioned reconstruction for real-to-sim studies. Finally, we release EnvDreamer-20k, a dataset of 20,000 validator passed environments with task programs, scene graphs, trajectories, and metadata to support reproducible benchmarking.