Human-guided robot policies improve precision tasks without retraining
PreferenceFlow: Test-Time Guidance of Flow-Matching Robot Policies from Human Interventions
Robotics
Summary
Robots often make mistakes when doing real-world tasks because their behavior was learned in different conditions. The authors present PreferenceFlow, a way to improve robot actions during use by learning from human corrections rather than needing explicit rewards or retraining. They compare human interventions with robot attempts to teach the robot which actions are preferred. Tests on a robot doing precise assembly tasks show that this method helps the robot succeed more often without changing its original program.
What this means in practice
- •For robotics engineers: Adjust robot manipulation actions on the fly using human feedback without needing to retrain the core control policy.
- •For human-robot interaction designers: Incorporate real-time human preferences to guide robot behavior during complex tasks for improved cooperation and efficiency.
Authors
Yiqi Tang, Diyuan Shi, Runze Li, Donglin Wang
Abstract
Flow-matching policies can represent complex robot behaviors but remain susceptible to local errors under distribution shift at deployment. Many reinforcement learning approaches to policy improvement require reward signals that are difficult to specify or obtain in real-world manipulation. We present PreferenceFlow, a framework for improving a pretrained flow policy at test time without environment rewards or updates to the base policy. Human intervention chunks are paired with robot chunks generated from the same initial conditioning state to train a preference model. During infer- ence, we adopt the QGF sampling update, replacing its value gradient with the preference gradient evaluated at an estimated clean action. A gradient-cap loss penalizes excessive gradients on intervention pairs, while a zero-gradient loss discourages guidance near actions from expert demonstrations. On four real-world precision insertion tasks with a Franka robot, Pref- erenceFlow achieves a mean success rate of 90.5%, compared with 69% for the frozen policy. Ablations support the roles of gradient regularization and correctly ordered preference labels in the evaluated settings. These results demonstrate the utility of human interventions as local preference supervision for guiding frozen generative robot policies.