Robotic control improved using masked action reconstruction
Robotic Spatiotemporal Editing: A Unified Interface for Controllable Action Generation via Masked Action Reconstruction
Robotics
Summary
Robots often need to perform tasks in ways that depend on where, when, and how they move, but controlling these actions precisely is hard. The authors treat different ways of guiding robot motion, like points and key poses, as partial clues to fill in the full action path. They created a system that can guess the missing parts of a robot's movement plan to match these clues, making robot actions easier to edit and control. Their approach works well both in simulations and real robots with low errors in position and rotation.
What this means in practice
- •For robotics engineers: Generate and edit robot arm motions using partial spatial or temporal constraints for precise task control.
- •For manufacturing automation teams: Improve robot action planning accuracy in assembly or manipulation tasks with flexible user-defined constraints.
Authors
Yanbang Li, Bu He, Yi Tian, Yu-Chiao Huang, Chenyu Zhang, Ruoqu Chen, Yuanzhe Liu, Yuxiang Gao, Mengdi Xu
Abstract
Generative policies can synthesize robot behaviors, yet controlling and editing them through diverse spatiotemporal references remains challenging. In the real world, human intent requires robot behaviors that satisfy constraints on where, when, and how actions are executed. Our key insight is that spatiotemporal control references, such as keypoints, flows, orientations, and keyposes, can be viewed as partial specifications of the same temporal Cartesian action space. We introduce the MAP (Masked Action Space) interface to unify these action references and formulate controllable manipulation as an action inpainting problem. Our Inpainting Policy (IP) reconstructs missing action dimensions of the end-effector, combining progress-guided alignment with multi-mask training to support unseen action-constraint combinations. We further adopt a training-free RePaint variant that iteratively inpaints the action during denoising to reduce control error. On ManiSkill and LIBERO-10, our policy achieves average task success rates of 92.67% and 64.67%, respectively.Real-world experiments demonstrate spatiotemporal control with average positional and rotational errors of 44.3 mm and 7°, respectively, showing that our policy can use diverse references to guide executable robot behaviors.