Embodied models fix errors faster with spotter and vision language models
Spotter: Let the Embodied Model Lead, and the VLM Reflect for It
RoboticsArtificial Intelligence
Summary
Robots that try to do tasks often fail and don’t know how to fix themselves. The authors found that current robot models rarely correct errors by retrying or reading a description of what went wrong. They designed Spotter, a system where the robot keeps acting but a language-capable AI watches, spots mistakes, fixes them, and lets the robot continue. This method improved robot success on tests, working faster than previous approaches that rely on the language AI for every step.
What this means in practice
- •For robotics engineers: Improve autonomous robot task success by integrating spotter to detect and correct errors dynamically.
- •For industrial automation teams: Reduce downtime by deploying vision-language models to monitor and correct robotic mistakes during operation.
- •For home automation developers: Enhance home robots’ reliability with spotter’s error spotting and correction running alongside embodied models.$Commercial implications: Enables consumer home robots to self-correct errors, making products more dependable and attractive to buyers.
Authors
Long Li, Qichao Zhao, Yue Yang, Fan Xu, Zhe Wang, Alan Wee-Chung Liew, Chao Qu, Heng Tao Shen, Shirui Pan
Abstract
Current embodied models do not respond to their own failures, although what just went wrong could inform a small adjustment on the next attempt, the kind of reflection behind the gains of thinking in language models. We test whether they can repair a known error, which requires producing a correction and judging whether it is right. Stopped at a failure and allowed to retry, they seldom repair it through their own randomness or from a language description of the error, and best-of-N selection cannot pick the successful candidate after a failure. We attribute this to training only on successful demonstrations and to inputs too narrow to show what went wrong, and conclude that reflection must come from a vision-language model (VLM), which takes in far more information, such as the episode history and text, and is more general. Prior VLM-led work has the VLM plan every step and invoke the embodied model as a tool, placing the VLM on the critical path. We propose Spotter, which reverses the roles: the embodied model leads and executes continuously, while the VLM runs in parallel, monitors through a lightweight local screener, intervenes only when an error is detected, reflects on and corrects it, and returns control. We run Spotter with Qwen and with GPT as the VLM, and both improve the embodied models; with GPT, Spotter improves Cosmos Policy and $π_{0.5}$ by 5.6 and 7.5 percentage points on RoboCasa, and raises $π_{0.5}$ from 47.2% to 57.0% on the Hard setting of RoboTwin 2.0 and from 53% to 83% on a real robot. Because the VLM steps in only when an error is confirmed, a successful episode with Qwen takes only 13 to 16 s longer than with the embodied model alone and about 70% less time than with a VLM-led baseline using the same model. Our code is available at https://github.com/zqc3117/Spotter.