Diagnosing Compositional Generalization in Sequential Robot Tasks
2026-07-31 • Robotics
Robotics
AI summaryⓘ
The authors studied how robots can learn to follow instructions made of familiar parts combined in new ways without needing to see every possible instruction combination. They found that understanding the different types of changes in instructions helps know when limited training data is enough. Their experiments show that training on a well-chosen small set of tasks that covers important relationships can lead to good performance on new tasks. They also discovered that adding just one new example per task can greatly improve success, especially when tasks depend on each other.
Sequential robot manipulationCompositional generalizationInstruction-space coverageOut-of-distribution (OOD)Action predictionTraining dataset sparsityDependency coverageTask generalizationFinetuningRelational structure
Authors
Yixiao Wang, Cheng-En Wu, Lingfeng Sun, Pengcheng Wang, Xiang Ji, Boyuan Liang, Guojian Zhan, Masayoshi Tomizuka
Abstract
Sequential robot manipulation requires policies to execute novel combinations of familiar instruction components. However, collecting demonstrations for all possible instruction tuples is combinatorially expensive, while sparsely covered datasets often fail under out-of-distribution recombination. This paper studies compositional generalization through the lens of instruction-space coverage. We decompose the generalization gap into three sources: \textit{marginal instruction shift}, \textit{instruction-compositional shift}, and \textit{context--action shift}. This decomposition allows us to diagnose when sparse training coverage is sufficient, and what structure the training set must preserve for reliable action prediction. Our results show that exhaustive tuple enumeration is unnecessary: a structured subset, as small as one quarter of the full task space, can recover strong out-of-distribution performance when it covers action-relevant dependencies. We further find that sparse training often fails due to instruction steering rather than missing low-level skills; finetuning only one demonstration per task improves OOD success from \(0.4\%\) to \(54.7\%\). For semantically dependent tasks, effective coverage must capture relational structure rather than only factor diversity. These findings suggest that efficient robot data collection should prioritize dependency coverage in instruction space over exhaustive task expansion. More results are available in the supplementary material. Project website: https://yixiaowang7.github.io/Diagnosing_Compositional_Generalization_Robot_Page/.