RoboAgent: Chaining Basic Capabilities for Embodied Task Planning
2026-04-09 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors address the challenge of teaching a robot (or agent) to plan and act in a complex environment by breaking down big tasks into smaller, easier steps. They created RoboAgent, a system that uses a single Vision-Language Model to handle different abilities that each focus on parts of the task, making reasoning clearer and easier to control. They train this model in multiple stages, including copying expert plans, learning from its own mistakes, and improving through feedback, while also using extra data to handle varied situations. Tests show their approach works well on standard benchmarks for robot task planning.
embodied task planningVision-Language Modelsmulti-turn interactionlong-horizon reasoningcapability-driven planningbehavior cloningDAgger trainingreinforcement learningrobotics benchmarksenvironment simulator
Authors
Peiran Xu, Jiaqi Zheng, Yadong Mu
Abstract
This paper focuses on embodied task planning, where an agent acquires visual observations from the environment and executes atomic actions to accomplish a given task. Although recent Vision-Language Models (VLMs) have achieved impressive results in multimodal understanding and reasoning, their performance remains limited when applied to embodied planning that involves multi-turn interaction, long-horizon reasoning, and extended context analysis. To bridge this gap, we propose RoboAgent, a capability-driven planning pipeline in which the model actively invokes different sub-capabilities. Each capability maintains its own context, and produces intermediate reasoning results or interacts with the environment according to the query given by a scheduler. This framework decomposes complex planning into a sequence of basic vision-language problems that VLMs can better address, enabling a more transparent and controllable reasoning process. The scheduler and all capabilities are implemented with a single VLM, without relying on external tools. To train this VLM, we adopt a multi-stage paradigm that consists of: (1) behavior cloning with expert plans, (2) DAgger training using trajectories collected by the model, and (3) reinforcement learning guided by an expert policy. Across these stages, we exploit the internal information of the environment simulator to construct high-quality supervision for each capability, and we further introduce augmented and synthetic data to enhance the model's performance in more diverse scenarios. Extensive experiments on widely used embodied task planning benchmarks validate the effectiveness of the proposed approach. Our codes will be available at https://github.com/woyut/RoboAgent_CVPR26.