Robot audits probe decisions to reduce failure in one shot tasks
Audit Before You Commit: Locating Belief Failures in Active Identification for One-Shot Manipulation
Robotics
Summary
Sometimes robots need to check a few times before doing a single important action to avoid mistakes, like tapping before inserting a peg. The paper finds that deciding when the robot has enough information depends on two things: the robot's beliefs must still include the true situation, and the model predicting failures must reflect actual problems. The authors test these ideas on simulated and real robot tasks, finding that small errors in sensing or execution can cause failures but fixing these helps a lot. Their method also shows patterns that help identify where the robot's understanding breaks down.
What this means in practice
- •For robotics engineers: Improve reliability of robot actions like insertion by auditing belief and failure models before committing to a single action.
- •For manufacturing automation teams: Use auditing techniques to detect and fix sensor or model errors that cause failures in automated assembly tasks.
Authors
Mohamed Abouagour, Byung-Cheol Min
Abstract
A robot that probes a few times before one irreversible action, such as tapping a surface before inserting a peg, must decide when the evidence is enough to commit. We argue that this decision rests on two conditions that existing methods do not separate: the belief must still cover the truth in the coordinate that decides the action, and the failure model that scores actions must track realized failure. We audit both conditions separately, offline and with ground truth, on a deployed probe-then-commit pipeline: a particle belief, a scenario failure score, and one commit. On simulated insertion, more taps sharpen the belief while the truth leaves its support on 16.9% of episodes and the failure score turns optimistic by 0.31. Conformal calibration restores coverage but not the decision: confidently wrong instances still pass a confidence gate. The audit's signatures instead point at the observation model, where a hand scan finds a 2.1 mm error in the tap boundary. Correcting that one number cuts failure from 0.354 to 0.112 on untouched instances and transfers unrefitted to a second engine, while in a third engine the same audit suggests an execution-model mismatch instead. Across seven task families in three engines, a few probes at a fixed executor reduce miss or failure. On a physical arm inserting a tool into a rigid pocket by touch, the gain and the audit's two conditions reproduce, and replaying the recorded taps under an injected model error shows the audit's signature on real data. Additional materials are available at https://sites.google.com/view/auditbeforeyoucommit.