SceneEvolve improves 3D scene reconstruction with smaller models
Learning to Code the World: Harness Evolution for Compositional 3D Reconstruction
Computer Vision and Pattern Recognition
Summary
Creating editable 3D scenes from images is hard because it involves understanding shapes, objects, and their positions. The authors developed SceneEvolve, a method that represents scenes as code commands which a small AI model can use and improve over time. By letting a stronger AI teach the smaller one how to better process scenes, they made the smaller model more accurate and faster than bigger models. This way, complex 3D scenes can be reconstructed more efficiently without constantly retraining the AI.
What this means in practice
- •For software developers in 3d graphics: Enable compact AI models to reconstruct detailed 3D scenes from images faster and with higher accuracy than large models.
- •For robotics programmers: Improve scene understanding by using evolved executable code to represent environments for robotic perception and interaction.
Authors
Haoyu Yao, Chenguo Lin, Ruize Du, Yadong Mu
Abstract
Reconstructing editable and compositional 3D scenes from a real image requires jointly reasoning about geometry, object structure, and spatial relationships. We introduce SceneEvolve, a framework that represents 3D scenes as executable code and improves reconstruction by evolving the harness around a small open-source agent. SceneEvolve combines SceneAct, a compositional semantic action interface built on parameterized primitives, hierarchical composition, and semantic spatial relations, with SceneHarness-RSI, which leverages a stronger teacher agent to iteratively evolve the harness using feedback from development-set reconstructions. Throughout harness evolution, the model parameters, action semantics, and execution backend remain fixed. The resulting harness provides a reusable reconstruction procedure that enables the small open-source model to reconstruct unseen scenes without parameter updates or teacher assistance. Across two in-domain datasets and one out-of-domain dataset, SceneEvolve with the comparatively compact Qwen3.8-27B model improves all four evaluated reconstruction metrics while reducing mean inference time by 47.0% relative to the base model. More notably, despite relying on a substantially smaller model, SceneEvolve achieves 32.3% higher reconstruction quality than the large proprietary GPT-5.6-Sol model, while reducing mean equivalent deployment cost by 59.8%. Furthermore, the harness evolved using Qwen3.8-27B also generalizes to Kimi-K3, achieving comparable reconstruction performance to GPT-6-Astra. The project page is https://haoyu-agent.github.io/SceneEvolve.