Dexoak improves robot hand motion capture retargeting with object contact
DexOAK: Object-Aware Kinematic Retargeting
Robotics
Summary
Robots often struggle to copy human hand movements precisely, especially when interacting with objects. The authors created DexOAK, a system that helps robots recreate human hand gestures while paying close attention to how the hand touches and holds objects. It uses a learning method that remembers object positions and adjusts the robot hand to avoid collisions and keep proper contact. Their method works better than earlier approaches and works well even when trying new tasks or objects.
What this means in practice
- •For robotics engineers: Generate more accurate robot hand motions from human demonstrations for tasks involving object manipulation.
- •For robotics hardware developers: Use DexOAK references for open-loop replay to collect successful robot manipulation data on physical robot hands.
Authors
Federica Spinola, Robert Katzschmann, Cordelia Schmid
Abstract
We introduce DexOAK, a learning-based object-aware kinematic retargeting framework for generating contact-consistent robot hand references from human motion-capture data. DexOAK first minimizes an interaction-graph Laplacian objective over a joint hand-object graph, so the robot reproduces the demonstrated object-relative interaction geometry rather than the human hand pose alone. These per-frame solutions form seed retargeted references, from which a feed-forward network conditioned on human keypoints and object point clouds learns the retargeting function. Object-aware finetuning then refines the network under contact, non-penetration, and self-collision losses that position the fingers on the demonstrated contacts and reduce artifacts destabilizing physics-based refinement. Once trained, the network transfers to unseen data by inference alone, without per-frame optimization. DexOAK references improve dynamic retargeting over the state-of-the-art across bimanual tasks with rigid and articulated objects. On the Inspire hand, success rises from 39.5% to 48.4% on OakInk-v2, and ADD-AUC rises from 62.7% to 77.9% on ARCTIC. Ablations trace the gains to object-aware kinematic retargeting and the amortized network refined with contact losses. Trained on OakInk-v2, it transfers to ARCTIC zero-shot and nears in-domain performance after fine-tuning. On hardware, 46/56 open-loop replays succeed across five tasks, enabling dexterous robot data generation from human data. Visualizations and interactive results are at: https://fspinola.github.io/dexoak/.