Bimanual robot control improves by aligning vision and physics knowledge
GRASP: Global-Local Representation Alignment and Spatial-Physical Grounding for Bimanual Robotic Manipulation
Robotics
Summary
Controlling robots with two hands is tricky because small mistakes in joint movements cause big problems when interacting with objects. The paper’s authors created a method that teaches robots better by focusing on how their hands interact with objects and considering their physical movement limits. They use smart training guides from other AI models to help the robot understand global and local aspects of the task. This makes robots better at long and complex tasks both in simulations and in real life.
What this means in practice
- •For robotic automation teams: Improve dual-arm robot performance on complex assembly tasks by training with physics-aware action alignment and focused vision guidance.
- •For vision system engineers: Design vision modules that help robots better identify and interact with target objects using learned attention queries without extra inference cost.
Authors
Zeyuan Ren, Guanxun Zhao, Qi Li, Hao Wang, Wenlong Liao, Keyi Luo, Xijing Zhang, Zhiyuan Zhang, Jingye Cui, Qibin Gu, Zhichen Wang, Yucheng Hu, Liqiao Xia, Nan Bu, Zhengde Ma, Rankun Li, Xue Yang, Wende Xie
Abstract
Vision-Language-Action (VLA) models that directly regress joint-space actions avoid the deployment-time cost of inverse kinematics. However, their training objectives remain misaligned with task success, as joint-space losses ignore the unequal physical consequences of joint errors and visual conditioning lacks explicit focus on hand-object interaction regions. To address these issues, we propose a training-phase prior injection framework for fine-grained bimanual manipulation. First, a physics-grounded flow-matching loss maps the implicit clean-action estimate at every noise level through differentiable forward kinematics (FK) and supervises the resulting Tool Center Point (TCP) pose and inter-arm relative pose, with a bounded noise-level weighting that compensates gradient decay. Second, we introduce a global prediction query group distilled from a frozen DINOv3 teacher through future-frame feature prediction, together with a local interaction query supervised by Grounding DINO at the attention level. Both teachers are removed at inference, and the deployment pipeline is that of a standard joint-space flow policy, adding only the learned query pathway. Simulation and real-robot experiments on long-horizon bimanual tasks show that the two priors are complementary across different deployment settings. The full model excels in in-distribution deployment, while the distilled query branch is preferable under strong visual randomization.