RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the challenge of improving depth accuracy in monocular 3D object detection, where current depth models lack precision at the object level. They propose RefineAny3D, a model that refines depth by visually aligning object boxes in the image instead of predicting exact depth numbers. This method treats depth errors as visual mismatches and uses categorical decisions guided by visual cues. Applied after detection, their approach improves performance across different detectors and adapts to new categories without extra training.
Monocular 3D object detectionDepth foundation modelsOpen-vocabulary detectorsDepth refinementVision-language model (VLM)Visual alignmentPost-hoc refinementChain-of-thought supervisionZero-shot generalization
Authors
Zhihao Zhang, Gengwei Zhang, Tianlong Chen, Xiaoming Liu
Abstract
Monocular 3D object detection spans two regimes: closed-set detectors operating within a fixed category vocabulary, and open-vocabulary detectors that localize arbitrary categories by leveraging depth foundation models for 3D geometry. We find that current depth foundation models, despite their strong zero-shot generalization, lack the object-level precision 3D detection demands: substituting a state-of-the-art depth foundation model for a strong detector's predicted depth degrades accuracy, even falling below the detector's own prediction. Rather than pushing detectors or depth models to be more accurate end-to-end, we treat object-level depth refinement as a stand-alone task and present RefineAny3D, a vision-language model that corrects depth without ever predicting a numerical value. Our key insight is that depth error has a direct visual signature in image space: when projected onto the image, a correctly placed box tightly encloses the object, while a too-far box projects too small and a too-close box projects too large. Depth refinement thus reduces to a visual alignment problem rather than a metric regression problem, which we instantiate by extending the VLM's vocabulary with action tokens that replace numerical depth output with categorical decisions, and by supervising the model on a large-scale chain-of-thought dataset that grounds each decision in explicit visual evidence. Applied as a single post-hoc step, RefineAny3D delivers consistent gains across closed-set detectors, open-vocabulary detectors, and 3D auto-labeling tools, and generalizes to novel categories, scenes, and cameras without retraining.