Wristband sensor estimates hand pose and finger force accurately

Beyond Gestures: Estimating Full Hand Pose and Contact Forces from Wrist-Worn Pressure Sensor Array

Human-Computer InteractionRobotics

Summary

Knowing how our hands move and how hard we press can help make computer games, virtual reality, and robots work better. The authors created a special wristband with soft sensors that can feel muscle movements and tendon shifts around the wrist. These sensor signals are turned into a detailed picture of finger positions and how much force each finger is applying. Tests show the wristband closely matches high-tech hand-tracking cameras and gloves, even when people use their hands naturally and with objects. This wristband could work together with cameras or other gadgets to help computers understand hand movements and forces better, even if the hand is hidden.

What this means in practice

  • For virtual reality developers: Create VR experiences that track finger positions and forces using only a wrist-worn sensor for more natural interaction.
  • For robotics engineers: Provide robots with rich tactile data from human demonstrations by estimating hand forces with a wearable wrist sensor.
  • For physical therapy device makers: Design wrist-worn devices that monitor patients’ full hand movements and finger forces during rehabilitation exercises at home.$Commercial implications: Enables selling wearable therapy aids that track hand recovery remotely using wrist-based sensing.

Authors

Svetoslav Kolev, Lingni Ma, Michael Goesele, Renzo De Nardi, Jakob Engel, Richard Newcombe

Abstract

Capturing hand motion and interaction forces is critical for interactive computing, VR, and high-fidelity tactile demonstrations for robot learning. We introduce a wrist-worn pressure-sensing wristband that recovers continuous full-hand pose and distributed contact force on a single wearable. The system consists of flexible capacitive sensor arrays around the wrist, which require no electrical skin contact, and a recurrent network that maps the resulting pressure signal to hand state. Our key insight is that muscle contraction and tendon displacement produce pressure patterns, which correlate strongly with hand pose and interaction force. To validate this, we collect synchronized recordings of wrist pressure, optical motion-capture hand pose, and tactile-glove interaction force, covering isolated finger motion, fingertip-force stress tests, and natural hand-object manipulation. On isolated single-user motion the wristband attains $4.6^\circ$ mean finger-joint MAE, and across four users manipulating everyday objects it estimates per-finger contact force at $R^2=0.57$, which an external pose signal brings up to $0.75$. We see the wristband as one node in a constellation of everyday wearables -- e.g. paired with an egocentric camera -- adding the contact force that vision cannot observe and taking over when the hand is occluded.
View PDFOpen arXiv