ARSTAG creates robot training data from images and instructions
ARSTAG: An Agentic Real2Sim2Real System for Task-Specific Robot Data Generation
Robotics
Summary
Robots often need lots of hand-coded setup or human guidance to learn new tasks involving objects. The authors present ARSTAG, a system that takes a single picture and a simple language instruction to automatically create training data for robots. It builds a virtual scene, generates robot actions, and creates varied examples to help robots learn better. Their tests show that robots trained with ARSTAG's data can perform different manipulation tasks with good success rates in real life.
What this means in practice
- •For industrial robot programmers: Generate custom robot training data from single images and instructions to speed up programming new tasks without manual data collection.
- •For home automation developers: Create task-relevant robot data from simple inputs to improve household robots handling diverse objects and arrangements.
Authors
Bowei Li, Yuner Zhang, Changliu Liu
Abstract
Adapting visuomotor policies to new manipulation tasks often requires substantial manual engineering or teleoperated data collection. Simulation can provide task-specific data at scale, but constructing the scene, designing expert behavior, and configuring data generation still require significant per-task effort. We present ARSTAG, an agentic Real2Sim2Real system that turns a single RGB image and a natural-language instruction directly into robot policy-learning data. A hierarchy of language agents constructs a task-scoped simulation scene, generates robot-feasible demonstrations, and expands the training distribution through task-consistent randomization, while a coordinator agent manages cross-stage feedback and recovery. Across seven manipulation tasks spanning grasping, placement, and stacking, the ARSTAG-generated demonstrations enable sim-to-real transfer of three visuomotor policy architectures to a dual-arm robot, with pi0.5 achieving an average real-world success rate of 74.6%. Ablations show that task-consistent randomization substantially improves robustness, and policy performance increases with generated dataset size. Project webpage: https://boweili666.github.io/ARSTAG/.