Hand motion model improves anomaly detection in two-handed assembly
Hand-Aware Transition Modeling for Bimanual Procedural Anomaly Detection
Computer Vision and Pattern Recognition
Summary
Detecting mistakes during two-handed assembly tasks is tricky because some fixes might look strange alone, while other actions seem normal but break the work order. The authors created a method called HACT that watches each hand’s moves and learns what’s expected over time. It spots unexpected actions by comparing them to both their meaning and timing within the task. Tested on different power-tool assembly tasks, HACT detected errors more accurately than other techniques and worked well even when task order changed.
What this means in practice
- •For industrial assembly teams: Monitor and detect errors in two-handed tool assembly lines by tracking hand actions and ordering to reduce rework and improve quality control.
- •For robotics engineers: Incorporate hand-aware temporal models to improve robot perception and interaction monitoring in collaborative bimanual assembly tasks.
Authors
Di Wen, Jimmy Weissert, Luc Maria Scherrer, Cedric Zöllner, Kailun Yang, Ruiping Liu, Yufan Chen, Jiale Wei, Junwei Zheng, Kunyu Peng
Abstract
Procedural anomaly detection in bimanual assembly requires judging each hand action against the execution so far. A corrective action may look unusual in isolation, while a visually plausible action can violate the order of the procedure. We present HACT, a transition model over predicted per-hand events. A role-preserving history keeps the concurrent responsibilities of both hands, and a marked temporal point process assigns each observed transition a semantic and temporal surprisal. A supervised evidence head and a two-state filter convert these surprisals into per-hand anomaly posteriors. A recovery-aware protocol on predicted events and participant-disjoint folds reports the recovery false-positive rate at an operating point selected on validation participants. On two bimanual power-tool procedures HACT has the highest AUPRC and F1 among the compared methods and the fewest recovery alarms. Applied without retraining to a different assembly order of the same product, it retains the highest AUPRC and F1. The source code is available at https://github.com/Kratos-Wen/HACT.