Summary
Robots learning to perform complex surgical tasks often struggle because they get little feedback when they fail, making it hard to know what part of their attempt was good. The authors introduced a method that uses a visual language model to identify which phase of the surgery the robot reached and when, giving more detailed guidance than just success or failure. This helps the robot learn from partial progress and improves performance on simulated and real tasks. Their approach showed better success rates than existing methods that only use a single overall score. This means robots can learn surgical manipulation more effectively even when rewards are sparse.
sparse-reward reinforcement learningsurgical manipulationvisual language modelphase feedbacktask phasescredit assignmentrobot learningsimulation and hardwaretrajectory-level supervisionCLIP
Authors
Wanli Liuchen, Fangyuan Wang, Bin Li, Anqing Duan, Yunhui Liu, Peng Zhou, David Navarro-Alarcon
Abstract
Sparse outcome feedback limits what robots can learn from unsuccessful attempts at complex manipulation. Failed multi-stage surgical attempts can contain grasps, lifts, or transfers worth reusing. In sparse-reward reinforcement learning, terminal rewards collapse such attempts to the same outcome, while scalar vision-language model (VLM) ratings reveal neither what progress merits credit nor when it occurred. We introduce phase-and-first-arrival feedback: one VLM query per recorded episode identifies the furthest visually verified task phase and when that phase is first reached, allowing the learner to reuse partial behavior and localize credit. We instantiate it in SurgPhaseBench, a phase-structured suite spanning rigid and deformable tasks, and evaluate it in simulation and hardware. Across five simulated tasks, our method reaches 75.2% mean success, compared with 52.1% for a reward based on Contrastive Language-Image Pre-training (CLIP) using the same visual input; the advantage persists when only the feedback representation changes. On hardware, the same record supports autonomous block picking and slip recovery. Together, these results show that trajectory-level visual supervision can preserve partial progress while providing the temporal credit needed for sparse-reward control.