3D large language models often miss key 3D evidence in scene questions
Do 3D Large Language Models Really Use 3D Evidence?
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
3D large language models (3D-LLMs) are designed to answer questions about 3D scenes, but they sometimes give wrong or unsupported answers because they don't fully use the 3D information available. The authors created UNICORNS, a new test set to check how well these models rely on real 3D evidence. They found existing models perform worse on challenging questions needing careful use of 3D data. To fix this, they developed HORN, a training method that helps models better understand when to trust and use 3D evidence, improving their accuracy on tough questions without hurting normal performance.
What this means in practice
- •For augmented reality developers: Improve AR applications’ scene understanding by integrating more reliable 3D-LLMs that use true 3D evidence in spatial reasoning.
- •For robotics software engineers: Develop robot perception systems that better assess and question visual 3D information for navigation and interaction tasks using evidence-aware 3D-LLMs.
Authors
Yifei Zhao, Fanyu Zhao, Han Zhang, Yixuan Lin, Yuyang Hou, Bin Wu, Ruike Cao, Li Xiao, Yinsheng Li
Abstract
Recent 3D Large Language Models (3D-LLMs) have shown strong performance on scene question answering with extensive post-training. Yet, existing benchmarks rarely evaluate how 3D-LLMs respond to referential ambiguity, insufficient scene evidence, or false presuppositions, obscuring whether their responses are grounded in 3D evidence. In this work, we introduce UNICORNS, a benchmark for evaluating the reliability of 3D-LLM responses across four evidence conditions. UNICORNS contains 11,387 questions across 557 ScanNet scans, covering eight challenge categories alongside answerable controls. A response passes only if it satisfies the required response behavior and contains no scene claim unsupported by the available evidence. Evaluation of six open-sourced 3D-LLMs reveals a pronounced performance gap: even the strongest zero-shot model achieves 61.22% accuracy on answerable controls, but only 40.94% across the eight challenge categories. To address this gap, we further propose HORN, an obligation-aware post-training method combining group-relative task rewards, leaf-conditioned obligation rewards, and supervised control replay. Extensive experiments demonstrate that HORN reaches $75.1 \pm 1.0\%$ across the challenge categories and $77.2 \pm 0.8\%$ overall, outperforming all baselines without degrading answerable-control accuracy. Our findings enable a systematic evaluation and improvement of evidence-grounded 3D reasoning for 3D-LLMs (code: https://anonymous.4open.science/r/unicorns-6B38/).