Holo-Captioning: Toward the Text Equivalent of 3D Scenes
2026-07-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors introduce holo-captioning, a new task that means creating detailed text descriptions of everything in a 3D scene, like the objects, where they are, their features, and how they relate to each other. They made a big dataset of over 15,000 scenes to train and test models, and created HoloScribe, a special model that can describe 3D scenes accurately by focusing on each thing and their connections. They also designed a new way to score how well the descriptions match the scenes, and their tests show HoloScribe works better than previous methods. This work helps machines understand and explain complex 3D environments better using words.
3D scene understandingcaptioningstructured descriptionsemantic tagsspatial locationinstance relationshipsbenchmark datasetHoloScribe modelevaluation metricinstance linking
Authors
Kun-Yu Lin, Chengke Bu, Zhenguo Li, Kai Han
Abstract
This work introduces holo-captioning, a novel task that strives to seek the text equivalent of 3D scenes. As the initial step, we formulate holo-captioning as generating a structured textual description that comprehensively depicts all entities within a 3D scene -- including their semantic tags, spatial locations, attributes, and inter-entity relations. To tackle this challenging task, we first develop an effective captioning engine to produce detailed descriptions of individual entity instances and instance pairs, and contribute a large-scale benchmark comprising over 15K scenes for training and evaluation. Building upon this foundation, we propose HoloScribe, a novel model that features an instance-aware decoupled pipeline for generating structured holo-captions, and further incorporates anchor-aware instance linking to identify relational instance pairs. Additionally, we propose a comprehensive evaluation metric named HoloScore, and provide a human-curated test set to ensure reliable model assessment. Experimental results demonstrate that HoloScribe significantly outperforms state-of-the-art 3D dense captioners and 3D LLM generalists, underscoring the effectiveness of our approach. Project page: https://visual-ai.github.io/holocap/