Summary
In tricky situations where lighting or movement makes it hard to control things from afar, a new wearable device helps by understanding what your hand wants to do. The device uses special sensors on your arm and hand to read muscle signals and movements. The researchers created a smart system that focuses on important signals linked to finger actions, making the control very accurate even when your arm is moving freely. Tests showed it can recognize hand gestures, measure how hard you are gripping, and copy hand poses very well, helping with tasks like pouring or grabbing objects. This kind of technology can make controlling machines remotely easier and more natural.
teleoperationsurface electromyography (sEMG)inertial measurement units (IMUs)deep learninghand pose decodinggesture recognitiongrasping force estimationhuman-machine interfacemultimodal sensor fusionwearable technology
Authors
Jiaxuan Li, Yinshi Wu, Xiao Zhang, Hongyu Wang, Renzhen Le, Zhenzhi Ying, Liming Shu
Abstract
Under ubiquitous teleoperation environments with optically challenging conditions, an interface for tele-operated grasping that combines wearability with precise decoding of hand intentions (hand pose, gestures, and grasping force) is essential. Yet, existing interfaces often fall short in meeting these demands, compromising either the diversity of multiple intentions decoding or wearability. To address this, we developed a novel Multiple Intentions Decoding Human-Machine Interface (MI-DHMI) that integrates high-throughput surface electromyography (sEMG) sensors with hand-mounted and forearm-mounted inertial measurement units (IMUs). The developed interface is supported by a unified framework for simultaneous multiple intentions decoding. By employing multimodal deep learning and hardware design with a low noise floor, the decoding framework selectively focuses on the sEMG components that are genuinely associated with finger movements. This effectively reduces decoding errors caused by sEMG variability during unconstrained upper-limb motions, thereby significantly enhancing robustness. Even under unconstrained wrist and forearm motion, the interface achieves a gesture recognition accuracy exceeding 97%, grasping force estimation with $R^2 = 0.95$, and hand pose decoding consistent with the actual hand pose, outperforming baseline devices and algorithms. Ablation studies further validate the effectiveness of the proposed decoding framework. Finally, two online experiments were conducted to validate the device, demonstrating its superior performance in high-stability tasks, including a pouring task and object grasping. The developed interface provides a new solution of a fully wearable, multiple intentions decoding system, offering effective support for ubiquitous teleoperation and contributing to the advancement of human-machine interaction research.