Matching category names outperforms geometry for 3d object retrieval
Token-Budgeted Candidate Retrieval for 3D Scene-Grounded VLMs: Category Names vs. Pairwise Geometry
Computer Vision and Pattern RecognitionMachine Learning
Summary
This paper studies how virtual language models (VLMs) that work with 3D scenes decide which objects and relationships to consider first within a limited context size. The authors compare two ways of selecting candidate objects: using object category names versus using pairwise geometric relationships. They find that matching query phrases to category names consistently works better than relying on geometric relationships, especially when recalling many objects. The study includes experiments on popular 3D indoor datasets and shows that semantic information beats hand-coded geometry cues under a fixed token budget.
What this means in practice
- •For augmented reality developers: Improve object retrieval in AR applications by prioritizing category name matching over geometric relation computations to enhance real-time scene understanding.
- •For robotics navigation teams: Enhance robot object localization in indoor scenes using token-budgeted retrieval strategies focused on semantic matching rather than costly geometric comparisons.
Authors
Michał Zakrzewski, Mariia Shpir, Taras Rumezhak
Abstract
Under a fixed context budget, which objects and relations should a 3D scene-grounded VLM see first? On ScanNet indoor referring benchmarks (Dai et al., 2017), matching the query to object category names outperforms pairwise geometry scoring at every reported K. The gaps are largest at high recall. Analytic pairwise cues beat per-object cue scoring and query-agnostic kNN graphs, including on paraphrases. MiniLM and BGE-small still score higher on those splits. We evaluate token-budgeted candidate retrieval with axis-aligned boxes, a token-ratio metric, training-free baselines, two small encoders that match q to each object's category name, and QCRR (Query-Conditioned Relation Retrieval), a deterministic pairwise-geometry scorer. Across ScanRefer gt_val plus a custom 20% scan-id split of Nr3D and Sr3D (35,504 queries; not official ReferIt3D val), additive noun matching reaches R@10 0.864 on ScanRefer against QCRR's 0.626; MiniLM and BGE-small reach 0.911 to 0.913 on ScanRefer and 0.941 to 0.947 on Nr3D. A corrected 3DGraphLLM harness (native Acc@0.25 0.67 on the full graph, n=300 ScanRefer queries) shows that context selection changes downstream accuracy under a matched object budget (Random 0.44, QCRR 0.56, MiniLM 0.67, oracle 0.70). Semantic retrieval beats hand-designed geometry on that harness. We will release the protocol, retriever implementations, and per-query outputs so new selectors can be swapped in without rewriting the analysis.