Chunk aligned semantic distillation improves multi stage robot tasks
CASD: Chunk-Aligned Semantic Distillation for Multi-StageRobot Manipulation
RoboticsArtificial Intelligence
Summary
Robots often perform tasks that have multiple steps, but labeling the first step doesn't explain the whole process. To solve this, the authors created a method called CASD that looks at entire chunks of robot actions and figures out what the robot should be doing throughout those chunks. They use a vision and language model to break down demonstrations into stages and then teach the robot to predict these stages from what it sees and knows. This approach helps robots perform better on several benchmark tasks, even when the tasks change a bit from what they trained on.
chunk aligned semantic distillationmulti stage manipulationvision language modelrobot policysemantic targetstask segmentationLIBERO datasetrobot statepolicy conditioningdistribution shift
Authors
Tinghe Ding, Jiahao Li, He Wang
Abstract
An action chunk can span several stages of a manipulation task, yet a label for its first step describes only the current stage. We introduce Chunk-Aligned Semantic Distillation (CASD), which derives semantic targets for entire action chunks. An offline vision--language model segments demonstrations into described stages. Their occupancy within each action chunk determines a weighted semantic target, including transitions between stages. A CASD generator learns to predict this target from the current observation, robot state, and task instruction. We then freeze the generator and train a policy conditioned on its predictions. The semantic branch runs once per policy query, without online VLM calls or reasoning-trace decoding. Teacher matching on annotated LIBERO training episodes is above chance for both single-stage and boundary-crossing chunks. We evaluate three Fast-WAM variants and a DreamZero integration across four benchmarks, including distribution shifts on LIBERO-Plus. Compared with published references, IDM+CASD reaches 98.9\% versus 98.0\% average success on LIBERO, while Uncond falls below its reference. Joint+CASD reaches 93.0\% versus 90.6\% on RoboTwin 2.0, and DreamZero+CASD reaches a 47.9\% four-category MolmoSpaces manipulation average versus 40.7\%. Performance varies across backbone integrations.