Robotic system identifies and locates hidden tomatoes in clusters
Active perception for robotic harvesting: 3D reconstruction and localisation of tomatoes hidden within clusters in a Mediterranean greenhouse
Robotics
Summary
Picking tomatoes with robots is hard because tomatoes often grow in clusters that block some fruits from view. The authors created a step-by-step method that uses 3D scans and smart processing to find tomatoes hidden inside these clusters. Their system cleans up the 3D data, groups parts of the plant, and estimates where each tomato is, even if mostly covered by leaves or other tomatoes. Tests show it can find over 90% of tomatoes accurately and locate them precisely within millimeters.
What this means in practice
- •For agricultural robot developers: Use the system to detect and localize tomatoes hidden inside clusters for improved autonomous harvesting accuracy in greenhouse farming.
- •For precision agriculture equipment manufacturers: Incorporate the 3D active perception pipeline to enhance design of fruit-picking robots that deal with occluded crops in dense plant environments.
Authors
Fernando Cañadas-Aránega, Rowan Border, José C. Moreno, José L. Blanco-Claraco
Abstract
Automating robotic harvesting in intensive agriculture within Mediterranean greenhouses requires overcoming significant challenges related to the geometric complexity of plants and occluded fruits. Although existing literature offers solutions targeting crops that grow in isolation (e.g., apples, sweet peppers, or peaches), the fundamental challenge lies in cluster-growing vegetables, where fixed sensors mounted on robotic systems fail to detect fruits hidden behind the visible surface. To address this limitation, this study presents a comprehensive pipeline for the 3D reconstruction and precise localization of each fruit within a cluster, including heavily occluded instances. The proposed methodology is structured into five sequential stages: i) point cloud acquisition using the AgriSEE Next Best View (NBV) active planner; ii) stochastic noise filtering via Statistical Outlier Removal (SOR); iii) surface classification and segmentation using Region Growing (RG); iv) isolation and recovery of occluded fruits through Density-Based Spatial Clustering of Applications with Noise (DBSCAN); and v) 3D pose estimation (position and orientation). This approach extracts the complete cluster geometry, ensuring the reliable identification of partially hidden tomatoes. Evaluated across multiple scenarios with varying occlusion levels within a simulation framework rigorously validated against real-world conditions, the system achieves a precision exceeding 90\%, an average recall of 82.8\%, and a mean Intersection over Union (mIoU) of 80.7\%. Furthermore, it demonstrates high repeatability in centroid estimation with a Root Mean Square Error (RMSE) of merely 4.2~mm, verifying its technical feasibility and high accuracy for autonomous harvesting operations.