Geometry aware language model improves object placement in 3D scenes
GALA: Geometry-Aware Language Model for Controllable Object Arrangement
GraphicsArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Placing objects realistically in 3D scenes, like furniture in a room, is tricky because it requires understanding how objects fit, support, or contain one another. Existing methods often use simple shapes that miss these details. The authors introduce a new model called GALA that sees the full shape of objects to better decide where to put them based on text instructions. They also created a new dataset of various room types and furniture layouts to train and test their model, which outperforms older approaches. Their system can generate scenes step-by-step by planning, placing, and checking objects using natural language instructions.
What this means in practice
- •For 3d game developers: Automatically arrange furniture and objects in virtual rooms following detailed text instructions, improving realism and creativity in game environments.$Commercial implications: Enables new scene generation tools for game studios to create realistic and context-aware indoor environments faster.
- •For augmented reality app developers: Place virtual objects realistically in users' real-world rooms by understanding detailed spatial relationships from language commands.
Authors
Chucheng Xiang, Runze Wang, Zhi Deng, Ruchao Bao, Yuanwei Zhang, Yuxuan Xie, Hanliu Wang, Liangzhen Fei, Wenzheng Wu, Cheng Wan, Peifeng Li, Zhongyuan Liu, Ligang Liu
Abstract
Reasoning about fine-grained placement constraints, such as support and containment, is essential for realistic 3D scene creation. It remains a significant challenge for existing methods that rely on simplified object representations like oriented bounding box (OBB), which are blind to the detailed geometry. To address this limitation, we introduce GALA, a geometry-aware language model with explicit point cloud perception for controllable object placement. Our model performs autoregressively by predicting the 6D pose of one target object at a time conditioned on the scene context and placement instruction. We utilize a two-stage training strategy: the first stage aligns geometric and textual modalities through pretraining tasks, and the second stage fine-tunes the model for placement. To facilitate this task, we also curate a multi-source scene dataset covering diverse room types and fine-grained furniture arrangements, which will be open-sourced for community benefit. Experiments on our benchmark show our model significantly outperforms baselines without explicit geometric reasoning. We further develop a multi-stage plan-place-verify agent pipeline that uses the model for text-driven scene generation. Project Page: https://xccelephant.github.io/paper/gala/