Proprioception improves robot failure analysis and timing detection
ProTracer: Proprioception-Guided Failure Diagnosis in Robot Manipulation
RoboticsComputer Vision and Pattern Recognition
Summary
Robots sometimes fail to complete tasks, and knowing why and when this happens is important. The authors created ProTracer, a system that uses both what the robot sees and what it feels through its own sensors to find out not only if a failure happened and what kind, but also the exact moment the problem started. ProTracer works without needing extra training by combining sensor data into clear descriptions that a vision-language AI can understand. They also made a new test set called FailTime to check how well systems can pinpoint failures in robot actions.
What this means in practice
- •For robotics engineers: Detect the precise moment and cause of robot task failures using combined sensor and vision data for better debugging and correction.
- •For manufacturing automation teams: Enhance failure detection and categorization in industrial robots by integrating proprioceptive signals with visual inputs without retraining AI models.
Authors
Chang Dong, Mehdi Hosseinzadeh, King Hang Wong, Lingqiao Liu, Francois Fraysse, Feras Dayoub, Minh Hoai Nguyen
Abstract
This paper presents a comprehensive framework for robot manipulation failure analysis that includes binary failure detection, failure categorization, explanation generation, and the additional capability of failure onset localization, which aims to identify the earliest moment at which a robot execution deviates from a valid task-completion trajectory and is ultimately followed by task failure. To address these tasks, we propose ProTracer, a training-free framework that leverages existing Vision-Language Models (VLMs) together with proprioceptive signals for failure analysis. Our method uses proprioceptive dynamics to identify temporally informative action boundaries and converts richer robot-state signals into structured natural-language descriptions that can be jointly analyzed together with visual observations by the VLM. This design combines the temporal precision of proprioceptive signals with the multimodal reasoning capabilities of modern VLMs without requiring additional model training. We further introduce FailTime, a benchmark with synchronized visual and proprioceptive observations for evaluating conventional failure diagnosis tasks as well as failure onset localization. Experiments demonstrate that ProTracer achieves strong performance across both conventional failure diagnosis tasks and the newly introduced failure onset localization task, highlighting the importance of proprioceptive reasoning for fine-grained temporal failure analysis.