Calibrated vision language model improves cluttered object grasping success
Calibrated Probabilistic Obstruction Reasoning with Vision-Language Models for Grasping in Clutter
Robotics
Summary
Picking up a specific object from a messy pile is tricky because sometimes other things are blocking it. The authors created a method called CPOR-Grasp that carefully considers different ways the scene might be arranged and how certain we are about which items are blocking others. By combining vision, depth, and shape information, their method better decides whether to pick up the target object now, remove something in the way first, or wait. This approach leads to much more accurate and reliable grasping in cluttered environments.
What this means in practice
- •For industrial robot operators: Enable robots to better decide whether to grasp a target object or move blockers in cluttered storage or assembly lines.
- •For warehouse automation teams: Improve reliability and efficiency of robotic picking systems for disorganized inventory by considering uncertainty in object occlusions.
Authors
Thanh-Tuan Tran, Ngoc-Chien Chu, Thanh Nguyen Canh, Nak Young Chong, Nguyen-Viet Ha, Xiem HoangVan
Abstract
Retrieving a target from clutter requires deciding whether to grasp the target, remove a blocker, or defer. Existing methods typically commit to a single obstruction graph or removal strategy, ignoring uncertainty across alternative scene interpretations. They also rely on miscalibrated vision-language model (VLM) predictions and can produce pairwise obstruction relations that are jointly inconsistent. Moreover, current approximations provide no guarantees about the impact of discarded hypotheses on the final decision. We propose CPOR-Grasp, a calibrated probabilistic obstruction-reasoning framework that propagates uncertainty from pairwise evidence to action decisions. CPOR-Grasp calibrates and fuses VLM, depth, and amodal-mask cues to estimate obstruction probabilities, induces a distribution over valid obstruction graphs, and marginalizes over these graphs to compute the likelihood that the target is accessible or that a given blocker should be removed. To make inference tractable, it retains only the highest-probability graphs and derives a total-variation bound on the discarded probability mass, enabling certified decisions, adaptive stopping, and principled deferral. On synthetic and real UNOBench scenes, CPOR-Grasp outperforms state-of-the-art baselines. Calibration error decreases from 0.1416 to 0.0185 on the Gemini Robotics backbone, while graph truncation matches exact inference on 99.74\% of decisions using 56 times fewer graphs. In real-world experiments, CPOR-Grasp achieves a 77.8\% average success rate, surpassing SOTA baselines.