Egocentric video improves robot grasping by pinpointing contact spots
EgoContact: Evaluating Physical Intelligence with Egocentric Video
RoboticsComputer Vision and Pattern Recognition
Summary
Robots need to know exactly where to touch objects to grab or use them properly, but this is hard to teach. The authors created EgoContact, a large video dataset showing where people contact objects during real interactions. They used it to test how well robots can guess the correct contact points and found many existing methods struggle. Using EgoContact’s data to remember good contact spots greatly improved robot grasping success in both computer simulations and real life.
What this means in practice
- •For robotic system builders: Improve robot grasping accuracy by using EgoContact’s verified contact points as a memory bank for learned manipulation strategies.
- •For virtual environment developers: Enhance training scenarios by integrating realistic human-object interaction contacts to better simulate future robot manipulation needs.
Authors
Adam Deryło, Alexey Gavryushin, Daniel Cremers, Marc Pollefeys, Xi Wang
Abstract
Egocentric video has emerged as one of the most scalable sources of data for learning generalizable manipulation priors. Yet, despite its growing use in pretraining robotic policies, it has been underutilized as a validation tool for physical intelligence, a key component of which is understanding where to make contact with an object to achieve an intended interaction. We present EgoContact, a large-scale egocentric dataset for evaluating the generalizability and precision of learned manipulation priors through future contact-point prediction. EgoContact contains 22.5K manually verified interaction events, each with a pre-contact observation, semantic intent, and pixel-precise contact locations. We find that current methods often struggle to predict precise interaction contacts and sometimes fail to localize the target object in diverse, in-the-wild scenes, even when allowed multiple predictions. Furthermore, we demonstrate the usefulness of EgoContact's annotated interactions for robotic manipulation: simply using EgoContact as the memory bank of an existing affordance-transfer policy improves grasp success from 14.2% to 31.2% in simulation and from 16% to 58% on a real robot.