Visual robotic models learn to fix mistakes from few examples
OopsieRecover: Learning Transferable VLA Corrections from Semantic Interventions
Robotics
Summary
Robots that use vision and language to pick up objects can make mistakes when working in new environments. The authors show a way to help these robots learn to fix their mistakes by understanding the reasons behind errors instead of just memorizing fixes. Their system, called OopsieRecover, uses a small number of corrections and learns general rules that apply to new objects and scenes. This means robots can improve from fewer examples and adapt better to new tasks and places.
What this means in practice
- •For robotics engineers: Improve robot manipulation policies by enabling quick recovery from errors with minimal corrective training examples.
- •For industrial automation teams: Deploy robots that adapt repairs and corrections to varied objects and scenes without retraining for each new setup.
Authors
Arpit Bahety, Skand Peri, Sriniket Ambatipudi, Chen Xu, Haruki Nishimura, Masha Itkina, Roberto Martín-Martín
Abstract
Vision-Language-Action (VLA) models can perform a wide range of robotic manipulation tasks, but often fail during deployment. Teaching them to recover from these failures requires substantial corrective data: existing approaches learn to recover from failures by directly associating corrective actions with the observations in which they were demonstrated. For instance, DAgger-like fine-tuning can require tens to hundreds of interactions, yet the resulting policy may still fail to generalize the learned corrections beyond the demonstrated scenarios. We hypothesize that semantics can capture the underlying reasons for failures and their corrections, enabling policies to learn from only a handful of corrective demonstrations and generalize those corrections to novel objects, and scene variations. We propose OopsieRecover, a framework that uses abduction to infer the intent behind a small set of corrective demonstrations and encodes the inferred knowledge as dense spatial representations and conditional costs. These costs capture when a behavior is undesirable and steer the policy toward more desirable action regions. We experiment across 6 tasks and variations in object poses, scenes, and related tasks, and find that only a handful of corrective demonstrations can enable a VLA to overcome failures across a broad range of novel situations, without requiring new corrective data for each variation. Please refer to https://robin-lab.cs.utexas.edu/oopsierecover for videos.