Robot learns manipulation tasks from one demo using world feedback
World In-Context Learning for Agentic Robot Manipulation
Computer Vision and Pattern RecognitionRobotics
Summary
Robots usually learn how to do tasks by training on many examples, which ties the knowledge to specific conditions. The authors show a way for robots to learn tasks from just one example by trying out the task in a simulated world and using feedback from that world, rather than changing their internal programming. This lets the robot adapt better to new scenes and tasks without extra training. Their system called RoboScript can learn in simulation and then use this knowledge on real robots with good success.
What this means in practice
- •For robotics engineers: Deploy robot manipulation tasks learned from single demonstrations and simulation feedback across different scenes without retraining the robot’s core model.
- •For industrial automation teams: Implement robots that quickly adapt to new assembly or handling tasks by rehearsing in simulated environments and applying learned task memories directly.$Commercial implications: Enables production robots to be reprogrammed rapidly with minimal data, reducing downtime and customization costs.
Authors
Heng Fang, Zehui Guo, Yuehan Zhang, Mingyang Du, Dingkang Liang, Xiang Bai
Abstract
Trained action policies write manipulation skill into network weights, tying what is learned to the distribution it is learned from. In-context learning (ICL) avoids this coupling but has not matched trained policies, because a demonstration recorded in another scene carries no feedback from the world the robot currently faces. We observe that what an agent learns in context is semantic, recording task steps and object roles rather than visual appearance, so experience gathered in one world transfers to another. Based on this insight, we propose World In-Context Learning (WICL), a paradigm in which a frozen model starts from a demonstration, rehearses the task in a world such as a simulator, and writes the feedback into its context rather than its weights. We build RoboScript, a modular infrastructure that implements WICL for agentic manipulation. From a single demonstration, RoboScript learns each task through rehearsal in simulation, freezes the learned knowledge as a Task Memory, and deploys the frozen memory on new scenes with data isolation that ensures every deployment input is available on a real robot. On all 50 tasks of RoboTwin 2.0, RoboScript reaches 83.4% mean success across clean and randomized settings with one demonstration per task and no gradient updates, exceeding the strongest trained action policy by over 20 points. The method generalizes across visual distributions, improves through reflection, scales with the foundation model, and transfers to a physical dual-arm robot through cross-embodiment memory adaptation. With a world in the loop, a robot can learn a manipulation task in its context as well as it learns one in its weights. All code is available at https://github.com/ArcheOne-AI/RoboScript