Robots learn from experience by improving policies and orchestrators together
Co-Evolving Robot Orchestrators and Policies through Deployment
Robotics
Summary
Robots that use vision and language to act often struggle when facing new situations outside of what they were trained for. The authors propose a system called Robo-COP where the robot’s decision-making guide (orchestrator) and its action policy improve together as the robot works in the real world. Robo-COP collects examples from its own attempts, fine-tunes its policy to fix repeated mistakes, and only accepts improvements after verifying better skill performance. This approach helps robots get better over time by learning from their own experiences.
What this means in practice
- •For robotics development teams: Enable robots to improve their execution skills automatically by updating both action policies and orchestrators based on deployment experience.
- •For manufacturing automation engineers: Use self-improving robot systems to adapt robotic tasks on factory floors as conditions and requirements evolve without manual reprogramming.
Authors
Xilun Zhang, Maggie Wang, Erik Bauer, Hong-Xing Yu, Huang Huang, Jiajun Wu, Marco Pavone
Abstract
Vision-language-action (VLA) policies trained on large datasets are capable within their training domains, yet they still fail to generalize to the variety of situations a robot meets in real-world deployment. Agentic robot systems complement the policy with a vision-language model (VLM) orchestrator that learns when to call the policy, how to instruct it, and when to use scripted skills instead. However, because the harness is built around a frozen policy that has limited language steerability, the orchestrator can avoid the policy's failures but never overcome them. The policy becomes the bottleneck of the whole system. Fine-tuning the policy can remove this bottleneck, but updating it alone decouples it from an orchestrator tuned to its old behavior. We propose Robo-COP, in which the orchestrator and policy co-evolve during deployment. Robo-COP curates skill demonstrations from its own executions, fine-tunes the policy when this data can address recurring failures, and adopts each new policy only after it improves the skills it was trained for. Across ten simulated RoboLab tasks, Robo-COP raises mean held-out success from 64.8% to 73.8% over the same harness with a frozen policy, while fine-tuning on a fixed schedule without verification reaches only 65.8%. On three real-world tasks, Robo-COP raises held-out success from 38.3% to 50.0%. Robo-COP turns deployment into a self-improving flywheel in which robots learn by doing, with each improvement in execution producing better data for the next round of learning. Videos and code are available at https://robo-cop.pages.dev/.