Rafail detects robot task failures by tracking key object relationships
RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation
Robotics
Summary
Robots sometimes fail to complete tasks, and catching these failures quickly is important. The authors created RAFAIL, a system that watches how parts of a robot and objects relate during a task to spot when something goes wrong. It uses smart ways to focus on the important parts of the scene instead of getting confused by unimportant changes. This helps RAFAIL detect failures more accurately without needing examples of failures beforehand.
What this means in practice
- •For industrial robot operators: Detect execution failures in robot tasks by monitoring important object and gripper relationships in real time to improve reliability and reduce downtime.
- •For warehouse automation teams: Improve robot picking accuracy by identifying unusual interactions between objects and robots without needing failure examples during training.
Authors
Loris Schneider, Edgar Welte, Rania Rayyes
Abstract
Detecting failures during execution is essential for reliable robotic manipulation. Vision-language models (VLMs) can assess task outcomes semantically but add runtime computation, whereas out-of-distribution (OOD) detectors may respond to harmless scene variations rather than failure-relevant deviations. We introduce RAFAIL, a framework for detecting execution failures during robotic manipulation. RAFAIL identifies failures by detecting anomalies in task-relevant relationships between entities, such as a gripper and an object or an object and its target. By focusing OOD detection on relevant parts of the observation, RAFAIL reduces sensitivity to task-irrelevant scene variation. Offline, a VLM annotates successful demonstrations with task progress and relationship importance, which are used to learn point-cloud-based relationship representations without relying on policy-internal features. At runtime, relationship-specific OOD detectors evaluate these representations while relationship importance and task progress are predicted without VLM inference. RAFAIL requires no failure data and achieves 73.4% balanced accuracy across three real-world robotic manipulation tasks, outperforming the strongest evaluated OOD- and uncertainty-based baselines.