System selects best camera views for 3D CAD from occluded scenes
Look Where You Can: Active View Selection for CAD Reconstruction under Occlusion
Computer Vision and Pattern Recognition
Summary
Getting a good 3D model of an object usually means looking at it from all sides, but real objects are often stuck in places where you can't see every angle. The authors created a system called SightCAD that smartly picks the best angles you can actually see to make a 3D computer model of the object. Their system learns which views matter most and how to turn the images into accurate CAD code. This approach works better than just picking random or evenly spaced views, especially when parts of the object are hidden.
What this means in practice
- •For industrial designers: Generate accurate parametric CAD models of machinery parts embedded in cluttered or occluded scenes from optimized camera viewpoints.
- •For robotics teams: Enable robots to select feasible camera angles for reconstructing objects with partial visibility in factories or warehouses.
Authors
Kartik Bali, Mahish Guru, Yiderigun Borjigin, Alexandra Starostina, Christian J. Cyron, Roland Aydin
Abstract
CAD reconstruction methods assume a luxury reality rarely grants: unrestricted visual access to the object, photographed from any desired angle. Real objects, however, are scene-embedded, bolted against walls, wedged into corners, resting on floors, where the scene renders much of the view sphere unreachable and the remaining views unequally informative. We introduce \textbf{SightCAD}, a framework for parametric CAD reconstruction that treats view feasibility as a first-class constraint. In this work we consider objects from standard CAD benchmarks embedded in realistic indoor scenes with physically derived visibility constraints over a discrete view sphere. A learned view selector must choose $K$ feasible views for a vision--language model (VLM) that generates executable CadQuery code, scored by geometric fidelity of the executed solid. Because reward arrives only after discrete view selection, autoregressive generation, and CAD-kernel execution, we propose a joint training paradigm in which the view selector and the CAD-generation VLM are trained together against this reward. The learned selection policy departs sharply from random, uniform, and coverage-greedy alternatives, outperforming surface-area maximization (SA-max) by up to $6.4$ Intersection-over-Union (IoU) points across budgets $K\in\{1,\dots,5\}$. The full system surpasses strong external baselines on scene-embedded, occluded multi-view renders of DeepCAD and Fusion360 objects ($+21$ and $+17$ effective-mIoU points over the best baseline, respectively), as well as on test-time domain-canonicalized real images from the industrial T-LESS benchmark and on both synthetic and real images from the MP6D industrial metal-parts benchmark, while producing the highest rate of executable programs of any method compared (invalid-code rate ${\leq}1.5\%$).