Two-finger gripper control improves cross-robot task learning
Enabling a Unified Cross-Domain Representation for Two-Finger Gripper Manipulation via Interaction-Centric Modeling
RoboticsComputer Vision and Pattern Recognition
Summary
Robots with different shapes and cameras usually struggle to learn tasks from each other. The authors found a way to represent how two-finger robot hands interact with objects so that the robot’s learning focuses on what matters for the task, not on the specific robot or camera view. They use smart tracking and a new prediction method to make smooth control actions from video and language input. Their approach works well both in simulation and on very different real robots without extra training.
What this means in practice
- •For robotics engineers: Design robot controllers that transfer grasp and manipulation skills seamlessly across varied two-finger robot hands without retraining.
- •For warehouse automation teams: Deploy manipulation systems that handle objects with different robot arms and views without extra customization for each robot model.
Authors
Guanlin Li, Shifeng Bao, Yihan Zhao, Haitao Shen, Haoyang Li, Chen Zhao, Tong Yang, Jie Tang, Jing Zhang
Abstract
Achieving robust cross-embodiment generalization in imitation learning demands overcoming a critical representation flaw that inextricably entangles task semantics with hardware-specific visual geometry. We propose an interaction-centric framework that leverages the shared structure of two-finger grippers via a parameterized universal gripper abstraction, yielding a canonical gripper-frame representation. Given language and RGB-D observations, a VLM infers the subtask and grounds an interaction triplet (gripper, held, target), while SAM~2.1 tracks masks to reduce VLM queries. We design concise hybrid features that combine target/collision artificial potential fields for global guidance with segmented gripper-frame point clouds for local geometry, and use a Flow-Matching Transformer to predict smooth 7-DoF action chunks. Experiments in simulation and real-world tasks demonstrate that ours is the first imitation learning approach to simultaneously achieve competitive benchmark scores and extreme cross-embodiment/cross-viewpoint zero-shot sim-to-real transfer to completely distinct, heterogeneous robot platforms.