Wearable device accurately decodes hand movements for teleoperation
Wearable Multimodal Human-Machine Interface for Integrated Hand Intentions Decoding in Dynamic Teleoperation
Robotics
Summary
Controlling robotic hands remotely can be tricky when movements and conditions are unpredictable. The authors created a wearable system that reads electrical signals and motion sensors from the wearer’s arm and hand to figure out intended hand poses, gestures, and grip force. Their approach uses smart computer algorithms to focus on the most meaningful signals, reducing mistakes even when the wearer moves their wrist or forearm freely. Tests showed their system recognized gestures with over 97% accuracy and precisely estimated grip forces, making it reliable for tasks like pouring or grasping remotely.
What this means in practice
- •For robotics engineers: Develop teleoperated robotic hands with reliable, wearable interfaces for precise hand movement and force control in complex environments.
- •For prosthetics designers: Create advanced prosthetic hand controllers that decode multiple hand intentions to produce more natural and stable movements during daily activities.
Authors
Jiaxuan Li, Yinshi Wu, Xiao Zhang, Hongyu Wang, Renzhen Le, Zhenzhi Ying, Liming Shu
Abstract
Under ubiquitous teleoperation environments with optically challenging conditions, an interface for tele-operated grasping that combines wearability with precise decoding of hand intentions (hand pose, gestures, and grasping force) is essential. Yet, existing interfaces often fall short in meeting these demands, compromising either the diversity of multiple intentions decoding or wearability. To address this, we developed a novel Multiple Intentions Decoding Human-Machine Interface (MI-DHMI) that integrates high-throughput surface electromyography (sEMG) sensors with hand-mounted and forearm-mounted inertial measurement units (IMUs). The developed interface is supported by a unified framework for simultaneous multiple intentions decoding. By employing multimodal deep learning and hardware design with a low noise floor, the decoding framework selectively focuses on the sEMG components that are genuinely associated with finger movements. This effectively reduces decoding errors caused by sEMG variability during unconstrained upper-limb motions, thereby significantly enhancing robustness. Even under unconstrained wrist and forearm motion, the interface achieves a gesture recognition accuracy exceeding 97%, grasping force estimation with $R^2 = 0.95$, and hand pose decoding consistent with the actual hand pose, outperforming baseline devices and algorithms. Ablation studies further validate the effectiveness of the proposed decoding framework. Finally, two online experiments were conducted to validate the device, demonstrating its superior performance in high-stability tasks, including a pouring task and object grasping. The developed interface provides a new solution of a fully wearable, multiple intentions decoding system, offering effective support for ubiquitous teleoperation and contributing to the advancement of human-machine interaction research.