Adaptive modulation improves dexterous robot hand tracking from language commands
AdaDexTrack: Dynamic Modulation for Adaptive and Generalizable Dexterous Manipulation Tracking
Robotics
Summary
Controlling robot hands to follow language commands for complex tasks is hard because small errors and differences between humans and robots add up over time. The authors present AdaDexTrack, a system that constantly adjusts what the robot tracks and how it represents objects to reduce these errors. They trained a general tracker on many examples and then improved it with a modulator that fine-tunes the robot’s movements during execution. This approach works better in tests with new objects and movements and even performs well when transferred directly to real robots.
What this means in practice
- •For robotics engineers: Build systems for robots to perform precise, long-duration hand-object tasks following spoken or written instructions.
- •For automation integrators: Deploy dexterous robotic manipulators that adapt in real time when handling unfamiliar objects in manufacturing or logistics.
- •For assistive device developers: Create prosthetic or assistive robotic hands that better interpret user commands and adjust grip dynamically.$Commercial implications: Enables improved command responsiveness and object interaction for assistive robots, supporting commercial prosthetic technologies.
Authors
Jianibieke Adalibieke, Qianwei Han, Xueyi Liu, Yuzhe Qin, Li Yi
Abstract
Language is a natural way to command robots, but converting a single instruction into a long-horizon, contact-rich hand-object interaction remains challenging: synthesized references are noisy, human-to-robot retargeting introduces embodiment bias, and fixed-reference tracking lets small errors snowball. We address this with AdaDexTrack, a modulator-in-the-loop framework for language-conditioned manipulation tracking. A distilled generalist tracker serves as the skill carrier, while a tightly aligned modulator performs three feedback corrections: reference modulation (continual adjustment of what to track), object-latent modulation (online adaptation of the object representation to recruit suitable skills), and positional-target modulation (small state-dependent refinements for execution). The tracker is learned via large-scale specialist to generalist distillation on a corpus of language-conditioned hand-object trajectories; the modulator is trained with RL under the same task objective, ensuring tight coupling. Across large-scale evaluations, AdaDexTrack consistently outperforms prior SOTA on unseen-trajectory and unseen-object sets in both average tracking error and success rate, demonstrating robustness and generalization. We further show zero-shot sim-to-real transfer on real hardware, where adding the modulator yields substantial gains over a tracker-only variant. AdaDexTrack reframes language-conditioned dexterous manipulation as modulated tracking, replacing the open-loop, fixed-reference tracking with in-loop modulation that adjusts the reference, object latent, and positional target, yielding drift-resistant execution from noisy text references.