Robot training data selection shows limited gains over full datasets
CuraEval: Evaluating Demonstration Curation on Mixed-Skill Robot Data
RoboticsArtificial Intelligence
Summary
Training robots to imitate humans often uses examples from people with different skill levels, but choosing the best examples is tricky. The authors developed a way to fairly test how well different selection methods work compared to randomly picking demonstrations or using all data. They found that simpler measures like choosing shorter demonstrations can improve robot success sometimes, but more complex scoring methods did not consistently beat random or full data use. This suggests that straightforward approaches might be as effective as more complicated ones when training imitation policies.
What this means in practice
- •For robotics engineers: Improve robot policy training by selectively using shorter demonstration data to enhance task success rates in imitation learning.
- •For automation system designers: Evaluate data selection strategies for training robots that imitate human operators with mixed skill levels to optimize performance efficiently.
Authors
Jiuyi Xu, Meida Chen, Yang Ye, Yangming Shi
Abstract
Robot imitation learning uses demonstrations from operators of varying skill, but selecting training data remains challenging. Existing methods score demonstrations using smoothness, diversity, or estimated influence on policy performance, but differences in selection and evaluation protocols make it difficult to determine whether they outperform simple baselines. We introduce \textbf{CuraEval}, a framework combining mathematical analysis of demonstration scores with controlled comparisons against fixed size-matched random subsets and full-data training on mixed-skill robomimic data using Diffusion Policy and BC-RNN. We show that the evaluated PSD score depends only on episode length and squared position magnitudes, and produces nearly the same ranking as episode length on \textit{square} ($ρ=0.999$). With Diffusion Policy, selecting the shortest 50\% of demonstrations improves \textit{square} success from 70.7\% under random selection to 76.5\%. Compared with random selection, our adapted geometric-entropy LOO selector reduces success at 25\% retention from 94.7\% to 74.0\% on \textit{can} and achieves 41.7\% on \textit{square}, below the mean success rates of all three independently sampled random subsets (59.8--62.5\%). On the same 3 \textit{square} Diffusion Policy training seeds, length-based selection at 75\% retention improves success over full-data training from 76.7\% to 86.7\% under the original protocol, but lowers it from 80.5\% to 79.5\% after checkpoint reporting, rollout count, and evaluation seed change jointly. No evaluated curated condition significantly outperforms full-data training after Holm correction, although confidence intervals still permit useful gains. Code is available at https://github.com/jiuyixu25/CuraEval.