Robots track task progress by spotting errors and recovery states
Tracking Task Progress Through Errors and Recovery:A Benchmark and Recurrent Distributional Model
Robotics
Summary
Robots often struggle to understand if they are doing a task well, making mistakes, or fixing errors as they work. To help with this, the authors created ReSTEP, a new collection of robot task videos labeled with exact mistakes and stages. They also developed REMORA, a method that uses memory and smart prediction to better tell where a robot is in a task, even when it looks very similar or has just recovered from an error. Their tests showed REMORA tracks progress more accurately and consistently than previous methods.
What this means in practice
- •For industrial robotics teams: Monitor assembly line robots to accurately detect task failures and recovery in real time, improving efficiency and reducing downtime.
- •For robotics system integrators: Integrate progress monitoring in manipulation robots for better troubleshooting and adaptive control during complex tasks.
Authors
Chen Wang, Xinze Niu, Zihao Zhu, Keru Zhou, Yicheng Xu, Shihao Jin, Wenhao Wang, Bowen Jiang, Chang Zhou, Li Liu, Maoqing Yao
Abstract
Operational strategies for long-horizon robotic tasks require reliable evaluation to determine whether execution is progressing, stalling, regressing, or recovering from errors. Yet real- world robot datasets that capture error recovery and provide frame-level error annotations remain scarce. To address this gap, we introduce and open-source ReSTEP, a dataset of 3,156 episodes across two manipulation tasks with semantic stage labels and frame-level er- ror annotations. ReSTEP supports training and benchmarking progress monitors through successful execution, mistakes, and recovery. Existing progress estimators can struggle to distinguish visually similar task stages and maintain temporally consistent predictions while responding to genuine setbacks. We therefore propose REMORA, an online progress esti- mator that combines recurrent visual memory, distributional prediction, and causal peak selection. Memory retains execution history to help resolve visual ambiguity, while distribu- tional prediction represents alternative progress hypotheses and causal peak selection tracks a supported hypothesis over time. Together, these components enable consistent progress tracking that remains responsive to errors and recovery. Experiments on both ReSTEP tasks demonstrate that REMORA outperforms the state-of-the-art SARM baseline in progress ac- curacy and temporal consistency.