Multimodal model predicts and warns of robot manipulation errors early
MAIA: Anticipating Interaction Anomalies in Contact-Rich Manipulation via Multimodal World Models
Robotics
Summary
Robot arms that handle delicate tasks often run into problems during contact with objects. Most current systems only notice mistakes after they happen, which can be too late to fix easily. The authors created MAIA, a model that watches visual, movement, and force data to predict if things will go wrong before the problem becomes obvious. They also made a big set of robot task videos with labeled errors to test their model. Their system gives earlier and more accurate warnings, helping robots avoid failure more often.
What this means in practice
- •For robotics engineers: Predict upcoming failures in robot contact tasks to improve reliability by acting before errors happen.
- •For manufacturing automation teams: Use early anomaly warnings to reduce downtime and increase success rates in automated assembly lines.
Authors
Yi Qiao, Maolin Zheng, Haoran Sun, Lujia Dong, Wenzhao Lian
Abstract
Reliable contact-rich manipulation requires detecting execution deviations early enough for corrective action. Yet most monitoring methods are reactive, identifying anomalies only after they become observable. Anticipating such anomalies is challenging because visual observations only partially reveal physical interaction, while planned actions describe intended motion rather than their actual physical consequences. We address this problem with MAIA, a multimodal framework for anomaly anticipation. Its vision-action-force world model learns nominal interaction dynamics and predicts the visual and force responses of planned actions, while a lightweight anomaly head predicts impending interaction anomalies. To enable systematic evaluation, we introduce CRIA, a real-world multimodal dataset of 2,852 trajectories spanning 18 contact-rich manipulation tasks, with frame-level annotations of anomaly onset and evolution. Experiments on both seen and novel tasks show that MAIA achieves more accurate anomaly discrimination than representative baselines while providing earlier warnings, with bACC scores of 0.765 and 0.710 and median anticipation lead times of 28.0 and 25.1 steps, respectively. Moreover, using these predictions for anomaly-aware action selection improves end-to-end task success from 52.6% to 61.5% without retraining the underlying manipulation policy. These results highlight the value of modeling future physical consequences for proactive anomaly anticipation.