Robot agent explores neural skills for diverse task learning
SkillWeaver: Agentic Exploration over Neural Interaction Skills for Scalable Robot Data Generation
Robotics
Summary
Collecting real robot data is expensive and limited in variety, so the authors designed a system called SkillWeaver that teaches robots in simulated worlds by letting an AI agent experiment with learned robot skills. This agent decides what to do next, uses robot skills that can adapt based on the situation, and learns from the results to plan better actions over long tasks. By generating lots of varied training data autonomously, SkillWeaver helps improve robot learning to handle new objects and settings, and makes transferring knowledge from simulation to real robots easier.
What this means in practice
- •For robotics engineers: Create richer robot training data sets autonomously by letting AI agents explore and combine learned interaction skills in varied simulated environments.
- •For automation system designers: Improve generalization and adaptability of robot behaviors to new objects and tasks by using data generated from agent-driven skill exploration.
Authors
He Zhu, Lusen Zhao, Kwan Man Cheng, Su Li, Katerina Fragkiadaki
Abstract
Large-scale demonstrations have driven unprecedented progress in robot learning, yet collecting robot data through teleoperation is expensive and difficult to scale to diverse environments and long-horizon tasks. Simulation offers a scalable alternative, but existing data-generation pipelines often rely on open-loop controllers, scripted skill sequences, or task-specific programs. We introduce SkillWeaver, an agentic framework that autonomously generates robot experience by exploring over Neural Interaction Skills (NIS): reusable, parameterized, closed-loop policies that expose learned physical interaction capabilities to a reasoning agent. Given a task and a simulated environment, a VLM agent reasons about what to do next, invokes and parameterizes NIS to interact with the environment, observes their outcomes, and generates verification, reflection, and memory to guide subsequent exploration. We instantiate NIS as reinforcement-learned policies for closed-loop, contact-rich manipulation and organize exploration as verifier-guided tree search, enabling the agent to discover successful long-horizon behaviors without relying on predetermined execution pipelines. SkillWeaver scales autonomously to 39.1K demonstrations across 14.1K scenes, which we distill into visuomotor policies. Across simulation benchmarks and real-world manipulation, training on SkillWeaver-generated experience substantially improves generalization to novel objects, spatial configurations, tasks, and environments, and enables zero- and few-shot sim-to-sim and sim-to-real transfer. Our results suggest agentic exploration over neural interaction skills as a scalable alternative for robot data generation.