Robot grasping improves by recalling past events and picking views
Beyond the Current Scene: Event-Referential Grasping with Active View Selection
RoboticsComputer Vision and Pattern Recognition
Summary
Robots often need to pick up objects based on how people used them before, even if those objects aren’t visible anymore. The authors created a system called BeyondSCe that helps robots find and grasp objects by remembering past events and choosing the best camera angles to see hidden targets. This system uses existing pretrained models and works well without extra training. In tests, it successfully grasped objects more often than other methods, especially when targets were hidden behind obstacles.
What this means in practice
- •For industrial automation teams: Enable robots to grasp objects referenced by past human interaction, improving handling of occluded parts in assembly tasks.
- •For warehouse robotics operators: Increase robotic picking accuracy by combining event memory with dynamic camera viewpoints to reveal hidden items.
Authors
Hyunjoon Lee, Haebeom Jung, Eunsung Cha, Daeun Lee, Yu-Chiang Frank Wang, Jaesung Choe, Jaesik Park
Abstract
A robot that observes people interacting with objects should be able to carry out later requests that refer back to those interactions. Such requests may specify a grasp target by the role it played in a past event rather than by its name or appearance. Moreover, the target may no longer be visible when the robot is asked to act. We present BeyondSCe, a zero-shot robotic grasping system for this event-referential setting. Given the event history and the current scene, the system identifies the requested object or part and localizes it for grasping. If the target is occluded, it combines an event prior recovered from the history with current scene geometry to select camera viewpoints likely to reveal the target. The system uses pretrained models without additional task-specific training. In real-robot experiments with a single wrist-mounted RGB-D camera, it achieves grasp success rates of 76% and 77% for initially visible and occluded targets, respectively, compared with 40% and 55% for the strongest baseline in each condition. On four additional scenes with heavy occlusion, it increases grasp success rates from 75% to 95% while reducing the mean number of views from 3.35 to 2.20, compared with an active-perception baseline given the target's ground-truth 3D bounding box.