HandsOn platform tests dexterous robot hand skills across tasks
HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation
Robotics
Summary
Robots need to use their hands in many ways with different arms, hands, and objects to do useful work. The authors created HandsOn, a large simulation setup that tests robot hands on lots of tasks with different robots and objects, including soft ones. They collected many human demonstrations and designed tests to measure how well robots can learn new tasks and adapt skills. Their tests also find where robots often fail, helping make future robot hand control better.
What this means in practice
- •For robotics engineers: Evaluate and improve robot hand algorithms for handling various objects and tasks within simulated environments before real-world deployment.
- •For industrial automation teams: Test robotic dexterous manipulation in diverse settings to develop versatile robot hands that adapt to new objects and assembly tasks.
Authors
Tianwei Luo, Yiming He, Hongjin Wu, Ting Pan, Debin Ren, Haorui Guo, Junlin Yang, Yuxin Zuo, Zhehao Kou, Kaisen Yang, Zhuoqun Lai, Yi Ding, Jiale Yu, Yuchen Zhang, Zijie Qiu, Haozhan Li, Ning Ding
Abstract
General-purpose manipulation requires reliable behavior across embodiments, objects, and task demands. We introduce HandsOn, a composable simulation platform and benchmark for cross-embodiment dexterous manipulation. It supports five robotic arms, six dexterous hands, and approximately 200 rigid, articulated, and deformable assets in randomized scenes. The benchmark pairs 18,000 curated human demonstrations with 66 human-verified tasks: Core-50 supports demonstration-based learning and execution tests, while Transfer-16 evaluates asset and relation transfer, skill composition, and deformable manipulation without target-task training demonstrations. Qualitative real-robot rollouts illustrate sim-to-real transfer. Across diffusion, vision-language-action, world-action, and prompted VLM-based methods, the best trained multi-task policy achieves 29.1% Core-50 success. Without benchmark finetuning, GPT-6 using move_eef for palm-and-finger control achieves 5.2% on Core-50 and 1.9% on Transfer-16 over 660 trials. Trained-policy stage diagnostics expose failures in contact establishment, action coordination, and maintaining achieved goals, with additional sequencing errors in composed tasks. HandsOn provides a shared basis for investigating manipulation generalization and the limits of reliable physical interaction.