Visual motion improves wearable thumb gesture recognition across users
ViMoWear: Visual Motion-Guided sEMG-IMU Representation Learning for Subject-Independent Thumb Gesture Recognition
Human-Computer Interaction
Summary
Recognizing hand gestures using wearable sensors is hard because signals differ between people. The authors developed a way to teach a computer model about hand gestures using both wearable sensor data and videos of hand movements, but when the model is actually used, it only needs the wearable sensors. This approach helps the model understand gestures better for new users it hasn’t seen before. Their tests showed improvements compared to older methods and allowed the model to retrieve gestures without extra classifiers.
What this means in practice
- •For wearable device developers: Improve hand gesture recognition systems by training models with visual motion data but using only wearable sensors during use, enhancing accuracy across users.$Commercial implications: Enables more accurate and user-generalizable gesture-controlled wearable products such as smart gloves and prosthetics.
- •For augmented reality developers: Create gesture control interfaces that work reliably for new users without requiring cameras during operation, facilitating simpler hardware setups.
Authors
Wenjuan Zhong, Chenfei Ma, Kianoush Nazarpour
Abstract
Wearable sensing enables intuitive hand gesture recognition for human--computer interaction, augmented reality, and prosthetic control, yet subject--independent recognition remains challenging because wearable signals provide only indirect and highly subject-specific observations of hand motion. Although visual information can improve wearable gesture recognition, requiring it during inference increases sensing complexity and limits practical deployment. We propose ViMoWear, a visual-motion-guided framework that leverages synchronized 3D hand motion as training-only supervision while requiring only wearable sensing for gesture classification at inference. Specifically, Motion-Guided Cross-Subject Contrastive Learning (MGCL) promotes subject-robust representations, and Thumb-Aware Masked Motion Reconstruction (TMMR) preserves fine-grained motion information. The leave-one-subject-out experiments on a synchronized sEMG--IMU--pose dataset demonstrate consistent improvements over supervised baselines across multiple sensing configurations, while the learned representations also support classifier-free retrieval. The proposed training-only visual motion supervision improves the generalization of wearable representations to unseen subjects.