Robot uses language and vision to improve disassembly tasks
Vision-Language Grounded Task-Context-Aware Imitation Learning for Robotic Disassembly
Robotics
Summary
Robots often struggle to take apart complex objects because there are many ways to do it and they can look very different each time. The authors show that by giving robots instructions in language along with visual input, robots better understand which parts to work on and what to do next. This method helps robots handle different shapes and setups without needing detailed labels for every part. Their approach significantly increased the success rate of robots completing disassembly jobs.
What this means in practice
- •For industrial automation teams: Improve robots that take apart complex products by using language-guided instructions combined with vision to increase task success.$Commercial implications: Enables manufacturers to sell more reliable disassembly robots that adapt without custom object labeling for each product line.
- •For robotics software developers: Create imitation learning models that link spoken or written instructions with visual scenes to select appropriate manipulation skills.
Authors
Jeon Ho Kang, Igal Tamarkin, Ethan Niu, Ian Novales, Satyandra K. Gupta
Abstract
Real-world robotic disassembly requires long-horizon execution, where robots must perform ordered sequences of manipulation tasks across multiple parts within a single scene. Multiple valid task goals and diverse assembly configurations make it difficult for imitation policies to infer the intended skill from raw observations alone, particularly when training data cannot cover the combinatorial diversity of real-world configurations and part geometries. We show that incorporating task context through language alleviates these challenges by providing explicit structure for skill selection and associating language-specified tasks with their corresponding manipulation targets in the visual scene. The proposed framework combines hierarchical task selection with task-context-aware imitation learning to ground language instructions in spatial visual representations for robotic disassembly. The resulting framework generalizes across diverse connector geometries and assembly configurations without requiring explicit object annotations. Our method improves end-to-end task success by 35 percentage points over the baseline diffusion policy and by 75 percentage points over the previous task-context-aware baseline.