Octrees improve 3D language models for shape and text understanding
Octrees as an Explicit 3D Language
Computer Vision and Pattern Recognition
Summary
Understanding 3D shapes and language together is hard because current methods either mix up the shape details or weaken the model’s language skills. The authors made a system called OctLLM that keeps the detailed 3D shape information in a special format called octrees, which organizes 3D data efficiently. They also kept the language ability strong by adding new parts only for 3D data, without changing the original language parts. This approach makes the system better at describing and understanding 3D shapes while still being good at regular language tasks.
What this means in practice
- •For 3d modeling software teams: Enhance 3D shape generation and captioning accuracy by using octree-based tokens to better capture spatial structure in models.
- •For computer vision developers: Improve integrated text and image systems by maintaining language capabilities while adding 3D shape understanding through independent training branches.
Authors
Ran Dan, Si-Tong Wei, Pengfei Xiong, Wei Zhang, Yadong Mu, Peng-Shuai Wang
Abstract
Existing 3D large language models (LLMs) compromise on two fronts: they compress shapes into latent codebook indices or coordinate text, which removes spatial structure from what the model observes, and they acquire the 3D modality by fine-tuning the backbone, which overwrites its general language ability. We present OctLLM, which addresses both limitations. Geometry enters as an explicit 3D sequence of octree occupancy tokens. However, full octree sequences grow rapidly with depth; OctLLM therefore randomly empties penultimate-level nodes and omits descendants while preserving shape, yielding a shorter coordinate- and depth-anchored Sparse Octree (S-Octree) for position-aware mask-modeling generation and 3D understanding. On the other front, existing methods introduce a new modality with full fine-tuning or LoRA, but full fine-tuning is costly, LoRA limits 3D capacity, and both modify the language pathway. OctLLM instead adds 3D capacity in parameters separate from the pretrained ones: mesh tokens are routed through independent trainable branches in a subset of blocks while text and image tokens retain the frozen vision-language pathway, and the two streams interact through shared self-attention. It trains far fewer parameters than full fine-tuning, yet sets a new state of the art among unified multimodal LLMs, lowering image-to-3D FID by $17.4\%$ and raising render-grounded captioning by $28.7$ points over ShapeLLM-Omni, while matching the backbone on general language benchmarks.