Vision language action models improve robot grasping in agriculture
Robot Learning with Vision-Language-Action Models for Agricultural Manipulation
Robotics
Summary
Robots find it hard to pick fruits and vegetables because plants can look very different and cameras can see things from many angles. The authors worked on teaching robots to understand what to do using both images and language. They used a computer simulation to train robots on how to reach and grasp tomatoes and then tested their learning on a real robot arm. They found one robot policy worked better and stayed more reliable when conditions changed. This shows combining learning from language, vision, and actions helps robots handle tricky farm tasks.
What this means in practice
- •For farm equipment builders: Create robot arms for farms that use multimodal models trained in simulation for reliable fruit picking.$Commercial implications: Enables sale of advanced robotic harvesters that adapt to varying plant shapes using pretrained vision-language-action models.
- •For industrial robot programmers: Fine-tune general robotic policies with domain-specific simulated demonstrations for specialized tasks like agricultural grasping.
Authors
Hamid Hafezi, Javad Mohammadpour Velni, Mohammadreza Davoodi
Abstract
Robotic manipulation in agricultural environments remains challenging due to variations in plant geometry, occlusion, and camera viewpoints. Pretrained vision-language-action (VLA) and generalist robot policies offer a promising approach to addressing these challenges by learning manipulation behaviors from multimodal observations and language instructions. However, adapting these policies to agricultural tasks requires domain-specific demonstrations, which are costly and time-consuming to collect on physical robots. In this work, we develop a simulation-to-real robot-learning framework for agricultural reaching and grasping that integrates task-specific demonstration collection and pretrained policy adaptation. A MuJoCo-based agricultural environment is developed with diverse tomato plant structures to support demonstration collection and controlled evaluation. Expert demonstrations are collected through joystick teleoperation, with a low-level controller used to execute robot actions. Two pretrained generalist robot policies, Octo and $π_{0.5}$, are adapted using simulated demonstrations, followed by further fine-tuning on a physical Franka Research 3 manipulator using real-robot demonstrations. The adapted policies are evaluated under in-distribution, viewpoint/spatial-shift, and out-of-distribution conditions in simulation, followed by physical-robot experiments. The influence of proprioceptive information on policy performance is also investigated. Results show that $π_{0.5}$ achieves higher task-performance scores than Octo and exhibits less degradation under distribution shifts. These findings demonstrate the potential of integrating pretrained generalist policies with task-specific demonstration learning for agricultural manipulation.