Dextera converts one image into robot hand control policies
DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real
Robotics
Summary
Getting robots to use their hands like humans is hard because it takes a lot of time and work to teach them using real-world examples. The authors created DEXTERA, a method that starts with just a single photo and builds a detailed digital model of the scene. This model helps robots learn how to move and interact with objects through simulations and shows that the learned skills can be used by real robots without extra training. They found that by mixing simulated practice with some real experience, robots perform tasks more successfully.
What this means in practice
- •For robotics engineers: Create robot hand control programs from a single image to jumpstart dexterous task development without manual scene modeling.
- •For virtual reality developers: Generate interactive digital twins for VR-based robot teleoperation and training using minimal initial data inputs.
Authors
Jin Wu, Lianjie Yuan, Zeyan Sun, Yuanyuan Lei, Disi A, Bicheng Han, Fangzhou Xia
Abstract
Collecting real-world robot data for dexterous manipulation is costly and time-consuming. While high-fidelity physics simulators enable scalable data synthesis and policy learning, constructing deployment-ready digital twins manually remains labor-intensive, and residual visual, geometric, and dynamics gaps hinder reliable sim-to-real transfer. We present DEXTERA, an automated real-to-sim-to-real framework that transforms a single RGB image into deployable policies for dexterous manipulation across four unified stages: (1) single-image scene factorization into a static Gaussian background and interactive rigid or articulated assets with VLM-inferred physical parameters; (2) metric scene global alignment, object canonicalization, and morphology-balanced robot calibration; (3) scalable simulator task primitive construction, VR teleoperation, and object-centric trajectory synthesis; and (4) a shared multimodal policy interface supporting both imitation learning and reinforcement learning. We evaluate DEXTERA across 13 task-embodiment pairs, 2 dexterous robot platforms, and 6 policy architectures. Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency. Furthermore, simulation-only trained policies enable viable zero-shot real-robot deployment, while simulation-real co-training substantially improves mean physical policy success from 29.2% to 61.9% across diverse policy architectures.