Progress guided preferences improve robot learning without human labels
Progress-Guided Preference Generation for Aligning Generative Control Policies
Machine LearningArtificial IntelligenceRobotics
Summary
Getting robots to learn tasks well usually needs expert examples, which can be hard to gather. This paper shows a way to help robots learn by using their own previous attempts to figure out which actions made progress. The researchers created a method that doesn’t need people to label or judge the robot's actions, making learning more automatic. Their tests show this method helps robots do better on complex tasks that take many steps.
What this means in practice
- •For robotics developers: Enable robots to improve task performance automatically by learning preferences from their own experience without needing human feedback.
- •For autonomous system engineers: Create control systems that self-adjust using progress signals rather than manual reward design, speeding up deployment for complex tasks.
Authors
Tung M. Luu, Tri Ton, Duc M. Nguyen, Xuan Thanh Nguyen, Nguyen Gia Binh, Quang Tan Nguyen, Vu Duong, Daniel Sonntag, Duy M. H. Nguyen, Vien Anh Ngo, Chang D. Yoo
Abstract
Generative policies have achieved strong performance in imitation learning tasks, but their success is often constrained by the limited availability of expert demonstrations. Although preference-based fine-tuning can leverage suboptimal rollouts to improve performance, existing approaches heavily rely on expensive human annotations. In this work, we introduce Progress-Guided Preference Generation (PGPG), an unsupervised preference alignment framework that eliminates the need for human-in-the-loop supervision. Our method leverages ordered coverage alignment to measure sequential progress relative to a small set of expert demonstrations, enabling the automatic extraction of pseudo-preference labels from autonomously collected rollouts. Furthermore, we introduce key enhancements to improve the reliability of these progress-guided preferences, enabling effective fine-tuning of pretrained generative policies without requiring manually designed reward functions. Extensive experiments across a suite of challenging long-horizon manipulation tasks demonstrate that PGPG outperforms standard supervised fine-tuning baselines. Moreover, our conceptually simple framework is broadly applicable across multiple classes of generative policies, demonstrating strong versatility when integrated with both diffusion and flow-matching policies.