Magar improves assembly error correction with multimodal agents
MAGAR: Multimodal Agents for GUI-Based 3D Mechanical Assembly and Minimal Rework
Artificial Intelligence
Summary
Sometimes instructions for assembling mechanical parts look right but lead to mistakes. The authors created MAGAR-Bench, a set of test examples, and MAGAR, a system that tries to fix assembly errors by carefully deciding when and how to change the assembly steps. MAGAR improves accuracy by focusing on correcting only the necessary parts instead of changing everything. This helps strike a balance between fixing errors and keeping correct actions intact.
What this means in practice
- •For assembly line engineers: Improve automated detection and correction of errors in mechanical assembly processes using multimodal agent decisions.
- •For robotics software developers: Develop more effective recovery strategies in robot-guided assembly tasks by applying selective correction frameworks that minimize unnecessary part movements.
Authors
Jingxuan Wei, Jiahao Fu, Fuqiang Wang, Xiaoming Wu, Yuwen Liu, Kaiwen Wei, Ming Yang, Junnan Zhu
Abstract
A GUI command can be correct at the interface level yet still produce an incorrect assembly. An agent must identify the relevant components, respect their structural relationships, and determine which parts should move during recovery. We introduce MAGAR-Bench, a benchmark of 4,000 multimodal samples from 26 assembly families that separates Assembly Decision, Atomic Action, and Minimal Rework. For rework, the objective counts distinct moved parts within the validated recovery candidate set for each state, rather than the number of actions. We also introduce MAGAR, a selective correction framework built on a frozen policy obtained by supervised fine-tuning. Its key design matches correction granularity to output dependencies. Structured candidates revise assembly fields and rework scope, while complete action candidates keep native GUI primitives coupled with their parameters. A learned selector retains the original prediction unless a candidate passes a task-specific intervention threshold. On recorded states from held-out assembly families, MAGAR improves Overall, the task-balanced aggregate, from 37.50 to 41.39, with gains on all three tasks. Across 209 interventions, 70 correct an error and 22 introduce one. These results support task-specific correction while highlighting the remaining tradeoff between fixing errors and preserving reliable predictions.