Robot manipulation improves by learning actions across stages
CASD: Chunk-Aligned Semantic Distillation for Multi-StageRobot Manipulation
RoboticsArtificial Intelligence
Summary
Robots often need to perform tasks that happen in several steps, but labels usually describe only the current step, making learning harder. The authors propose a method called CASD that groups these steps into chunks and teaches the robot about the entire chunk using signals from a vision-language model. This helps the robot better understand transitions between stages and improves its ability to complete tasks. They test this approach on several robot benchmarks and show better success rates compared to previous methods.
What this means in practice
- •For robotics engineers: Train robots to handle complex multi-step manipulations by learning semantic meaning across task stages for improved accuracy.
- •For automation system integrators: Develop more reliable robotic manipulation pipelines that efficiently interpret task instructions without costly online vision-language model calls.
Authors
Tinghe Ding, Jiahao Li, He Wang
Abstract
An action chunk can span several stages of a manipulation task, yet a label for its first step describes only the current stage. We introduce Chunk-Aligned Semantic Distillation (CASD), which derives semantic targets for entire action chunks. An offline vision--language model segments demonstrations into described stages. Their occupancy within each action chunk determines a weighted semantic target, including transitions between stages. A CASD generator learns to predict this target from the current observation, robot state, and task instruction. We then freeze the generator and train a policy conditioned on its predictions. The semantic branch runs once per policy query, without online VLM calls or reasoning-trace decoding. Teacher matching on annotated LIBERO training episodes is above chance for both single-stage and boundary-crossing chunks. We evaluate three Fast-WAM variants and a DreamZero integration across four benchmarks, including distribution shifts on LIBERO-Plus. Compared with published references, IDM+CASD reaches 98.9\% versus 98.0\% average success on LIBERO, while Uncond falls below its reference. Joint+CASD reaches 93.0\% versus 90.6\% on RoboTwin 2.0, and DreamZero+CASD reaches a 47.9\% four-category MolmoSpaces manipulation average versus 40.7\%. Performance varies across backbone integrations.