Scale Up Strategically: Learning Compositional Generalization via Bias-Aware Evaluation and Data Collection for Robotic Manipulation
2026-07-23 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors studied how robots follow instructions by looking at which parts of the instruction the robot pays most attention to, like color or action words. They found that robots often rely too much on some features, especially color, and ignore others like verbs and size. To measure this, they created ways to rank which instruction parts dominate robot decisions. Using this insight, they improved robot learning by focusing training on the less-used parts, which helped the robot learn better with fewer examples. This shows their method can help robots understand instructions more evenly and effectively.
compositional generalizationpretrained policiesinstruction factorsFactor Dominance Rate (FDR)Factor Dominance Hierarchy (FDH)semantic componentsrobot instruction groundingbias in machine learningdata collection strategypolicy learning
Authors
Yu Qi, Zhang Ye, Xinyi Xu, Yuxuan Lu, Amitoj Sandhu, Boce Hu, Haojie Huang, Jonathan Tremblay, Lawson L. S. Wong
Abstract
Compositional generalization is essential for robot to follow diverse instructions. However, pretrained policies are known to take shortcuts, deferring to salient cues rather than grounding language. We introduce a diagnostic framework that localizes this failure to individual \textit{instruction factors}, \textit{e.g.,} reusable semantic components such as color, verb, object, size, and spatial attribute. Our framework formalizes instruction factor bias, the tendency of fine-tuned policies to over-rely on dominant factors as shortcuts, and quantifies it through two metrics: Factor Dominance Rate (FDR), capturing pairwise bias between factors, and Factor Dominance Hierarchy (FDH), aggregating these into a global ranking. Evaluation on six foundation policies reveals broadly consistent ordering, \textit{i.e.}, color $\geq$ object $\geq$ spatial $\geq$ verb $\geq$ size, with color dominant, and verb and size most under-grounded. We further show the diagnosis is actionable: a bias-aware data collection strategy that reallocates a fixed budget toward under-grounded factors outperforms baselines in simulation and on a real robot using half the demonstrations, thereby enabling more sample-efficient and generalizable policy learning.