SILICA: Repurposing Diffusion Priors for Joint Glass Segmentation and Depth Estimation
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionRobotics
AI summaryⓘ
The authors address the problem that regular depth sensors struggle to see transparent surfaces like glass, which causes issues for robots and 3D mapping. They propose SILICA, a new method that uses large text-to-image diffusion models to better detect and measure glass without needing special glass depth data. SILICA predicts both where glass is and how deep it is, improving safety and mapping accuracy. Their experiments show SILICA works well even in new places it has never seen before, beating previous methods by about 20%.
depth sensorstransparent surfacesmonocular depth estimationdiffusion modelsglass segmentationzero-shot transfer3D mappingautonomous navigationdatasetmachine learning
Authors
Tarun R, Anuj Verma, Laksh Nanwani, Sourav Garg, K. Madhava Krishna
Abstract
Standard depth sensors systematically fail on transparent surfaces, creating corrupted 3D maps and severe navigation hazards. While specialized hardware sensors can detect glass, they lack modularity and have extensive hardware dependencies. Consequently, learning-based monocular depth estimation has emerged as a compelling alternative. However, domain-specific glass-aware monocular depth estimators struggle with unfamiliar indoor layouts; restricted by the severe scarcity of real-world glass depth annotations, they fail to generalize zero-shot to new settings. This motivates us to explore whether the extensive priors of text-to-image diffusion models can enable generalizable perception of transparent surfaces. We introduce SILICA, a unified pipeline leveraging these priors to jointly predict glass segmentation and glass-aware depth. This mutual information exchange establishes a robust spatial hierarchy, entirely eliminating the need for paired real-world glass depth annotations. Subsequently, we use the predicted segmentation mask to explicitly filter incorrect glass depth points from standard sensors, recovering accurate metric glass depth for downstream 3D mapping and autonomous collision avoidance. Supported by our novel Mirage 18k dataset, extensive experiments demonstrate that SILICA achieves remarkable zero-shot transfer across diverse, unseen environments, outperforming state-of-the-art models by almost 20% and setting a new benchmark for transparent surface perception.