Vision language models guide multi arm robots picking fruit
From Vision to Harvest: Benchmarking Vision-Language Models for Multi-Arm Robotic Fruit Harvesting
RoboticsComputer Vision and Pattern Recognition
Summary
Picking fruit with multiple robot arms can be tricky because the robots have to work together without getting in each other's way. The authors created a test to see how well current vision-language AI models can plan these multi-arm fruit-picking jobs without any training on the specific orchards. They found that these AI models can make good plans right away, but still struggle with precisely figuring out 3D positions and avoiding collisions. This shows that while the technology looks promising for helping robots pick fruit, there are still challenges before it can be used in real farms.
What this means in practice
- •For agricultural robotics engineers: Generate multi-arm harvesting sequences using pretrained vision-language models in real orchard environments without additional training data.
- •For warehouse automation teams: Plan coordinated multi-arm robot tasks using vision-language models to improve efficiency in cluttered shared spaces.
Authors
Vrishan Inukollu, Adyan Zaman, Anvi Kudaraya, Carlos Lazcano, Yuankai Zhu, Stavros Vougioukas, Xiaofan Yu
Abstract
Multi-arm robotic harvesting offers a promising path to improve harvesting efficiency and reduce reliance on manual labor. However, practical deployment remains challenging because the system must generalize across diverse environments while efficiently coordinating multiple arms in a shared workspace. Existing methods often require substantial data collection in target environments or rely on simplifying assumptions that limit planning quality. In this work, we introduce the first comprehensive benchmark for evaluating pretrained Vision-Language Models (VLMs) on zero-shot multi-arm fruit harvesting planning. Our benchmark uses real-world apple and citrus orchard images and compares a VLM-based planning pipeline with a traditional perception-and-planning pipeline. The VLM pipeline directly generates harvesting sequences and waypoints for each arm, while a lightweight trajectory verifier checks for collisions. Our results show that frontier VLMs can generate effective multi-arm harvesting plans zero-shot, but a practical deployment remains limited by accurate 3D waypoint generation and collision-aware coordination. These results highlight both the promise and current limitations of pretrained VLMs for multi-arm robotic harvesting.