Joint modeling improves robot two-arm manipulation success rates
JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation
Robotics
Summary
Coordinated use of two robot arms is hard because each arm’s movement changes the shared space and affects the other arm. Most existing robot controllers predict actions without fully considering these future changes in 3D space. The authors propose JAMB, a method that predicts actions and future movements together, letting each prediction inform the other. Tests in virtual and real robots show JAMB achieves higher success and better adapts to cluttered scenes than other approaches.
What this means in practice
- •For robotics engineers: Improve coordinated control algorithms for robots using two arms to handle complex object manipulations in dynamic environments.
- •For manufacturing automation teams: Develop robot assembly lines where two-armed robots can better adapt to cluttered setups or unexpected object placements during tasks.
Authors
Chuyang Xiao, Peilin Meng, David Held
Abstract
Coordinated bimanual manipulation is challenging because the motion of either arm can alter the shared 3D scene and thereby affect the other arm. Yet most diffusion policies generate actions without explicitly modeling these future geometric consequences, while predictive variants typically use future state only as auxiliary supervision or fixed conditioning. We address this limitation by proposing JAMB, a diffusion policy that jointly denoises bimanual actions and future 3D point tracks. By allowing action and track hypotheses to evolve together within a shared Transformer, each can inform and refine the other throughout denoising. We further ground multimodal representations in a shared spatiotemporal coordinate system to facilitate geometry-aware interaction during joint denoising. We evaluate JAMB on diverse bimanual manipulation tasks in RoboTwin 2.0 and on a real-world robot, comparing it with action-only policies and alternative future-prediction approaches spanning different state representations and learning objectives. Across 16 simulation tasks, JAMB achieves an average success rate of 83.4%, outperforming the strongest baseline by 23.9 percentage points. On three real-world tasks, it outperforms the action-only and auxiliary geometry prediction methods by 50.0 and 21.2 percentage points, respectively. Beyond these performance gains, JAMB shows stronger generalization to cluttered scenes and out-of-distribution backgrounds than the evaluated baselines. Together, these results demonstrate the effectiveness of our joint action-motion modeling framework for coordinated bimanual manipulation. Our project website is available at https://jam-bimanual.github.io/