InfiniDream generates large photorealistic 3D worlds with text input
InfiniDream: Open-Ended Photorealistic 3D Worlds Through Procedural Generation
Computer Vision and Pattern Recognition
Summary
Creating detailed 3D worlds by hand is hard and time-consuming. The authors introduce InfiniDream, a method that uses language models and procedural algorithms to build realistic and editable 3D scenes automatically from text instructions. Their system can arrange many objects precisely, add animations, and produce diverse outputs based on different seeds. Tests show it improves on existing approaches in quality and variety.
What this means in practice
- •For game developers: Generate vast, realistic 3D game worlds automatically from text descriptions with customizable and diverse assets.$Commercial implications: Enables game studios to create richer environments efficiently, reducing manual asset creation time and costs.
- •For synthetic data teams: Create large-scale photorealistic 3D scenes with precise object placements for training computer vision models.
Authors
Yi-Chuan Huang, Hao-Jen Chien, Zheng-Hui Huang, Yu-Lun Liu, Kaipeng Zhang, Zhixiang Wang
Abstract
Procedural generation plays a key role in synthetic data creation and game development, offering greater controllability, editability, and diversity than static mesh representations. However, VLM-based code generation often produces relatively simple 3D assets, while sophisticated photorealistic procedural systems require extensive manual development. Existing VLM-guided procedural methods remain largely constrained by predefined factories and APIs, limiting their ability to create novel asset types. We introduce InfiniDream, a VLM-driven framework for open-ended procedural 3D world generation. First, we propose Symbolically Constrained Asset Generation, which leverages existing procedural algorithms to create novel, photorealistic assets while preserving editability and seed-based diversity. Second, we develop an end-to-end text-to-world pipeline that combines semantic scene planning with Deterministic Frustum-Guided Grid Placement to construct large-scale 3D worlds with precise object placement, procedural population, and animation. Extensive evaluations on 100 object and 20 scene generation tasks demonstrate improvements across four perceptual metrics. Diversity analysis, ablation studies, and human preference evaluations further validate the effectiveness of our framework. Project page: https://yichuanh.github.io/infinidream-page/