Robot task planning improves with modular vision and closed loop system
TrySee: Modular Perception-Enhanced VLM Grounding for Closed-Loop Robot Task Planning in Specialized Domains
RoboticsArtificial Intelligence
Summary
Robots use vision language models (VLMs) to understand images and plan tasks, but these models struggle in specialized areas. The authors present TrySee, a system that combines general VLM reasoning with specialized vision modules and robot components. This setup provides continuous feedback and adapts without needing retraining or large datasets. They tested it on various robot tasks, including real-world disassembly, showing it works well even in complex environments.
What this means in practice
- •For robotics engineers: Integrate modular perception with VLM reasoning to improve robot task planning in specialized industrial settings.
- •For automation system builders: Deploy adaptive robot planning systems that use replaceable vision modules without retraining for various domain-specific tasks.
Authors
Elias Hofele, Thomas Trapp, David Timmermann, Tristan Schnell, Georg Heppner, Arne Roennau
Abstract
Vision Language Models (VLMs) have shown impressive capabilities in high-level reasoning and task planning for robots. However, specialized domains remain a challenge for VLMs' image interpretation and grounding abilities. We introduce TrySee, a modular closed-loop robot task planning pipeline integrating general-purpose VLM reasoning with domain-specific perception and established robotics components. The system consists of an inherently reactive cyclic planning pipeline, augmented with domain-specific modular vision preprocessing, to provide online grounding. Finally, a dynamic preprocessing selection mechanism contributes to the scalability of the pipeline. The system requires neither VLM fine-tuning nor end-to-end demonstrations and adapts through replaceable perception modules and motion planning. The approach is evaluated on planning tasks across 5 widely different domains, including a real-world robot disassembly task. The field tests in a brownfield environment demonstrate the approach's practicality in integrating VLM planning into specialized robotic systems with limited domain-specific training data.