Multimodal teleoperation improves whole-body mobile robot manipulation
M3-Tele: A Unified Multimodal Teleoperational Framework for Compliant Whole-Body Mobile Manipulation
Robotics
Summary
Contact-rich tasks, like manipulating objects while moving, need careful coordination and sensing to work well. The authors developed M3-Tele, a system that lets operators control mobile robots using multiple types of senses—like vision, touch, and force—at once. M3-Tele helps robots interact physically with objects more smoothly and collects better synchronized data for learning later. Tests showed it reduces errors and unintended contact significantly and is user-friendly. Their experiments also suggest combining tactile and force sensing improves robotic control policies.
What this means in practice
- •For robotics engineers: Improve robot teleoperation interfaces for complex physical tasks by integrating vision, touch, and force sensing for better control and data collection.
- •For automation integrators: Enable more reliable mobile manipulation in industrial settings by reducing contact failures and improving compliance during teleoperated assembly and maintenance tasks.
Authors
Hengxiang Chen, Shenwen Deng, Yujian Ma, Gan Ma, Qiang Li, Nutan Chen
Abstract
Executing contact-rich tasks efficiently requires the seamless integration of whole-body coordination and physical compliance regulation. However, existing teleoperation and data-collection frameworks often overlook the joint consideration of multimodal perception and coordinated whole-body operation. This limitation can reduce the efficiency and quality of demonstration collection, thereby affecting the effectiveness of downstream policy learning. In this work, we present \textbf{M3-Tele}: A Unified \underline{M}ultimodal \underline{Tele}operational Framework for Compliant Whole-Body \underline{M}obile \underline{M}anipulation, enabling stable physical interaction and capturing aligned visual, tactile, force, and proprioceptive observations during task execution. Extensive experiments demonstrate that the proposed framework significantly improves contact-rich teleoperation performance. The proposed controller reduces the force tracking error from 4.132~N to 0.346~N, the contact loss from 2.46 to 0.02 events per trial and the tactile deformation error by 65\%. User studies across four mobile manipulation tasks also verify the reliability and usability of the proposed system. Furthermore, Diffusion Policy experiments highlight the value of joint tactile and force sensing.