Model separates 3d scenes into objects faster with less data
CoLa3D: Composable Latent 3D Decomposition
Computer Vision and Pattern Recognition
Summary
Breaking down 3D scenes into individual objects is hard, especially from just one image or mesh. The researchers developed CoLa3D, a method that looks at the whole scene at once and breaks it into object parts directly in a shared space, avoiding tricky steps used before. This method works faster and uses far less training data while creating consistent, realistic 3D object shapes that fit back into the scene properly. It does well on both made-up and real-world 3D data, preserving visible details accurately.
What this means in practice
- •For game developers: Generate consistent and detailed 3D object models from single images or scene meshes for immersive and interactive virtual environments.
- •For robotics teams: Improve robots’ understanding of cluttered 3D environments by decomposing scenes into accurate, reusable object models for navigation and manipulation.
Authors
Xudong Cai, Yongcai Wang, Deying Li, Chuanxia Zheng
Abstract
We introduce CoLa3D, a simple and effective model for composable 3D scene decomposition that, given a single whole-scene latent from either a single image or an existing scene mesh as input, directly outputs a set of object meshes with consistent spatial layout that can be assembled back into a coherent scene. CoLa3D introduces a novel paradigm for 3D scene decomposition within a shared whole-scene latent, unlike alternatives that perform per-object generation in canonical spaces and then glue the objects back together with fragile pose and scale estimation. The network is a lightweight, flexible, and scalable query-based transformer that accepts various types of queries, including 2D masks, 3D points, and learnable queries, and learns to decompose the scene into objects in a compact latent space. On both synthetic and real-world datasets, CoLa3D achieves state-of-the-art performance on composable 3D scene decomposition, substantially improving global scene-level consistency and object-level quality while running 2 times faster and using less than 1% of SAM 3D's training data. Visible depth on par with MoGe2 further confirms that latent decomposition preserves accurate visible scene geometry while producing physically complete, decomposable object meshes.