Preference-guided steering shifts robot behavior beyond known limits
PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviors
RoboticsArtificial IntelligenceMachine Learning
Summary
This work addresses the problem of changing how a robot behaves when desired actions are rarely or never seen before in its usual behavior. The authors propose a method called PrefPI, which uses relative preferences between robot actions to gradually guide the robot toward new behaviors. Instead of needing exact examples of correct actions, their method learns from comparisons of different attempts, making it possible to explore new behaviors beyond what the robot initially does. They demonstrated that PrefPI significantly improved a robot's ability to lift objects higher using only limited feedback.
What this means in practice
- •For robotics engineers: Guide robots to develop new, higher-performing behaviors with limited preference feedback rather than exact demonstrations.
- •For automation integrators: Adjust pretrained control policies to handle tasks with behaviors initially unseen during training using preference comparisons.
- •For industrial manipulators: Improve real-world robotic object handling capabilities, such as lifting heights, through preference-guided learning without extensive reprogramming.$Commercial implications: Enables selling enhanced robotic manipulation systems that adapt to new tasks with minimal human input.
Authors
Seungeun Rho, Wontaek Kim, Danfei Xu, Sehoon Ha
Abstract
We present PrefPI (Preference-Guided Policy Iteration), an iterative framework for steering pretrained generative robot policies using only relative preferences over self-generated trajectories. Unlike prior preference-learning methods that primarily sharpen modes already represented by the policy, we study steering beyond the initial effective support, where desired behaviors are rarely or never observed under the initial policy. Our key idea is to formulate preference learning as preference-conditioned generative modeling: preferred trajectories define a conditional distribution, whose density ratio with the broader behavior prior provides an implicit preference signal amplified by classifier-free guidance (CFG). Repeating this preference-conditioned modeling and guidance step yields a form of preference-guided policy iteration, turning incremental improvements toward previously inaccessible behaviors. Across diffusion policies and the PI0.5 flow- matching VLA in simulation and the real world, PrefPI produces substantial behavioral shifts with limited feedback. In particular, PrefPI increases object transport height from 10.7 cm to 19.8 cm on real hardware with only 150 preference-labeled trajectories.