Voxel programming improves 3D object placement with language models
VoxProg: 3D Spatial Reasoning through Voxel Programming
Computer Vision and Pattern Recognition
Summary
Figuring out where to place objects in a 3D space based on language instructions can be tricky because words often leave out details about positions and shapes. The authors developed VoxProg, a way for a language model to write small programs that work with detailed 3D grid views of object surfaces to understand and place objects correctly. By interpreting shapes and nearby surfaces explicitly, the system can better decide where and how to put objects based on the instructions. They tested their approach on a new benchmark of 71 tasks and found it outperforms other recent 3D placement methods.
What this means in practice
- •For robotic system developers: Enable robots to interpret complex spatial instructions by generating voxel-based programs for precise 3D object placement and manipulation.
- •For 3d video game designers: Create tools that automatically position objects in 3D scenes according to descriptive language inputs for immersive and contextually accurate game environments.
Authors
Zhichao Zhai, Menglei Zhao, Guikun Chen, Zihao Yan, Kanzhi Wu, Wenguan Wang, Jun Xiao
Abstract
Language models' abilities to interpret tasks and generate programs offer a promising basis for 3D spatial reasoning. Yet following spatial instructions in 3D scenes requires programs to construct geometric references that language leaves implicit. In this work, we introduce VoxProg, which formulates 3D object placement through voxel programming: a language model translates task interpretations into executable programs over local voxel views of scene surfaces. Because these views index surface geometry on regular grids with explicit coordinates and neighborhood relations, an evidence program can isolate task-relevant regions and boundaries directly from scene meshes, and a downstream pose program can measure the resulting geometry to compute a target $SE(3)$ pose. Visual and numerical feedback from execution lets the model inspect intermediate selections and candidate placements to iteratively revise its code. To evaluate open-ended placement across functional use, geometric fit, and contextual arrangements, we introduce the 71-task VoxProg-Bench benchmark along with an evaluation suite pairing perceptual Intent Alignment and 3D Physical Validity. Across VoxProg-Bench and external benchmarks, VoxProg consistently outperforms relation-based, keypoint-based, and mesh-based baselines.