Framework coordinates robot decisions for object retrieval with limited views
Selective Commitment for Language-Guided Object Retrieval under Partial Observability
Robotics
Summary
Finding objects with a robot can be tricky when the robot can't see everything clearly. The paper presents a system that helps a robot decide when to look more, move things, pick up objects, or give up. The system keeps track of what the robot might know about where an object is and uses smart planning to improve finding the object. Tests showed this system worked better than other methods in varied scenarios and could even recover from some mistakes during real robot trials.
What this means in practice
- •For warehouse automation teams: Enable robots to reliably find and pick specific items inside containers even when parts of the scene are not fully visible.
- •For assistive robotics developers: Improve robot assistance by letting robots decide autonomously when to gather more info or grasp objects based on language instructions.
Authors
Wonhee Koh, Sushil Samuel Dinesh, Hansol Ko, Shinkyu Park, Eungjoo Lee
Abstract
Language-guided object retrieval under partial observability requires deciding whether to gather more evidence, interact with the scene, grasp a candidate, or abstain. We present a closed-loop framework that coordinates these decisions for retrieving a target specified in relation to a reference container. The framework maintains a persistent joint belief over target identity, container relation, and presence through tracked-object, unobserved-target, and target-absent hypotheses. View-conditioned categorical VLM observations update this belief; conformal grasp eligibility and robot feasibility govern commitment, while finite-horizon belief-space planning selects information-gathering actions. Across five different scenarios, our proposed method succeeds in 19/25 simulation episodes versus 12/25 for the best-performing task-adapted baseline and is the only evaluated policy to achieve at least one success in each scenario. Ablations show that cross-view memory improves success under partial occlusion, while the full system does not consistently outperform simplified variants. Real-robot trials demonstrate closed-loop re-observation and autonomous recovery from injected grasp failures, while injected viewpoint failures end in false defer. Experimental results demonstrate the feasibility of coordinating evidence gathering and selective grasp commitment within a unified framework for retrieval under partial observability.