ScenePilot: Grow-and-Repair Policy for Text-Driven 3D Indoor Scene Generation
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors propose ScenePilot, a new way to create 3D indoor scenes from text descriptions by building the scene step-by-step and fixing problems along the way. They use a planning module to retrieve examples of room layouts and object groups to guide where things should go. After placing objects, a repair module corrects mistakes to keep the scene realistic and well-organized. They also created a special dataset called SceneReverse-17k to train this fixing process by showing how to undo common errors. Their approach aims to be more efficient and produce better scenes than methods that generate everything at once or require a lot of expensive fixing afterward.
3D indoor scene generationtext-driven synthesislayout priorshierarchical retrievalfunctional groupingreinforcement learningmultimodal repairSceneReverse-17kgrow-and-repair frameworkobject placement
Authors
Jiawei Zhang, Hongsong Wang, Pan Zhou
Abstract
Text-driven 3D indoor scene generation has advanced from dataset-bound layout modeling to open-vocabulary synthesis with large language and vision-language models. Yet existing methods remain limited: one-pass generators often yield geometrically invalid layouts, heavy post-hoc optimization is costly and unstable, and prompt-only planners lack reusable layout priors for functional grouping and object relations. We propose \textbf{ScenePilot}, a retrieval-augmented \textbf{Grow-and-Repair} framework that formulates scene generation as prior-guided incremental growth with learned rectification. Given a prompt, the Hierarchical Retrieval-Augmented Planning (HRAP) module retrieves room-, group-, and anchor-level layout priors to support functional group planning. A text-driven base generator then inserts object groups sequentially, while the Reinforcement Multimodal Repair (RMR) module performs lightweight local correction after each insertion and a final global repair after completion. To train this policy, we construct \textbf{SceneReverse-17k}, a repair-trajectory dataset built by perturbing high-quality 3D scenes in position, rotation, and scale, then using inverse operations as executable rectification targets. The policy predicts structured \emph{move--rotate--scale} actions from rendered views, scene state, retrieved priors, and edit history. By combining HRAP with RMR, ScenePilot offers an efficient alternative to one-shot generation and heavy full-scene optimization, improving physical plausibility, functional coherence, and controllability while preserving diversity.