Multimodal teleoperation improves robot task handling and safety
M3-Tele: A Unified Multimodal Teleoperational Framework for Compliant Whole-Body Mobile Manipulation
Robotics
Summary
Robots that handle objects and move around need to carefully sense and react to touch and movement to work well without causing damage. This paper presents a system called M3-Tele that helps operators control such robots by combining visual, touch, force, and body movement information. The system reduces errors in how the robot senses contact and force, making tasks smoother and safer. Tests and user studies show it is reliable and easy to use, and it also provides valuable data for teaching robots new skills.
teleoperationmultimodal perceptionmobile manipulationforce sensingtactile sensingproprioceptioncontact-rich tasksrobot controlpolicy learning
Authors
Hengxiang Chen, Shenwen Deng, Yujian Ma, Gan Ma, Qiang Li, Nutan Chen
Abstract
Executing contact-rich tasks efficiently requires the seamless integration of whole-body coordination and physical compliance regulation. However, existing teleoperation and data-collection frameworks often overlook the joint consideration of multimodal perception and coordinated whole-body operation. This limitation can reduce the efficiency and quality of demonstration collection, thereby affecting the effectiveness of downstream policy learning. In this work, we present \textbf{M3-Tele}: A Unified \underline{M}ultimodal \underline{Tele}operational Framework for Compliant Whole-Body \underline{M}obile \underline{M}anipulation, enabling stable physical interaction and capturing aligned visual, tactile, force, and proprioceptive observations during task execution. Extensive experiments demonstrate that the proposed framework significantly improves contact-rich teleoperation performance. The proposed controller reduces the force tracking error from 4.132~N to 0.346~N, the contact loss from 2.46 to 0.02 events per trial and the tactile deformation error by 65\%. User studies across four mobile manipulation tasks also verify the reliability and usability of the proposed system. Furthermore, Diffusion Policy experiments highlight the value of joint tactile and force sensing.