Digital-twin ar interface improves teleoperation for robotic hands
Now You Feel It, Now You See Me: Digital-Twin-based Teleoperation Interface for Dexterous Manipulation
Robotics
Summary
Controlling robot hands to do delicate tasks is hard because you can't feel the forces or see hidden parts during control. The authors developed an augmented reality interface that shows a virtual copy of the robot hands and objects, including force feedback as visual cues and special views to prevent hidden parts from blocking sight. This helps operators handle objects more precisely and safely by improving what they see and feel during remote control. They tested this approach in tasks that need careful finger control and dealing with blocked views.
What this means in practice
- •For robotics system integrators: Enable operators to remotely control robotic hands with enhanced visual and force feedback during complex object manipulation tasks.
- •For virtual reality interface designers: Incorporate occlusion-aware rendering and force visualization in AR interfaces to improve user performance in teleoperation and training scenarios.
Authors
Youngchan Shim, Kyutae Lee, JooYun Kim, Jaeseong Hwang, Harim Ji, Yongseok Lee
Abstract
Teleoperation is becoming increasingly important for collecting high-quality demonstrations to teach robots dexterous manipulation skills. For dexterous manipulation, bare-hand tracking provides a practical way to control robotic hands and demonstrate coordinated finger movements without gloves or exoskeletons. However, this type of teleoperation faces two key feedback limitations: a lack of force feedback and visual feedback of occluded region. The absence of force feedback hinders precise and safe manipulation, as operators must infer contact force visually rather than feel them directly. Occlusion by objects or other robot parts impairs the assessment of hand positions, approach distances, and pre-grasp configurations suited to the object's shape. To address these limitations, we propose a digital-twin-based augmented reality (AR) teleoperation interface that integrates bare-hand tracking, bimanual robotic hands, and virtual representations of the robot and task objects. The interface renders tactile measurements on robot hand meshes as visuo-force feedback. Furthermore, it offers a user-controlled virtual view panel or occlusion-aware transparency rendering to provide occlusion-mitigating visual feedback. We evaluated the interface through user studies involving Task 1 and Task 2, examining how force visualization and visual assistance support demonstration collection in tasks requiring careful force regulation and manipulation under occlusion.