Robot learns to pick specific objects in cluttered scenes
Towards Fine-Grained Object Manipulation: SAM3-Guided Visuomotor Policy with Persistent Memory Learning and Focused Visual Conditioning
Robotics
Summary
Picking out one particular object among many that look alike is hard for robots. This paper presents a new way for robots to remember and recognize exact objects even when other similar things are around. The method uses a special memory system and focuses on the object’s features combined with its location to guide the robot’s actions. The researchers tested their system on a set of 30 real objects and showed it could reliably find and manipulate the right item despite distractions. This approach helps robots be more careful and precise when handling everyday objects.
What this means in practice
- •For industrial automation engineers: Enable robots to accurately identify and handle specific items on busy factory lines where many similar parts are present.
- •For warehouse robotics teams: Improve robot picking systems to reliably select individual products from shelves with visually similar items amid clutter.
Authors
Haolong Meng, Fangbo Qin, Mengchen Bai, Houwu Wang, Cirong Liu, Shan Yu
Abstract
Fine-grained object (FO) manipulation requires robots to distinguish a specified FO from visually similar objects and execute actions reliably despite scene distractors. However, scene-level visual conditioning lacks explicit object selection, while category-level guidance cannot reliably distinguish FOs within the same category. We present a SAM3-guided visuomotor framework that addresses these challenges through persistent object memory and focused visual conditioning. First, we introduce FO Memory-driven SAM3 (FOM-SAM3), which learns reusable FO memory tokens from limited multi-view registration images while keeping SAM3 fully frozen. Through one-vs-rest learning, these tokens encode persistent memories for localizing target FOs and rejecting similar alternatives, which can be stored in a memory bank. Second, we propose Focused Spatial-Appearance Encoding (FSAE), which combines in-FO local appearance features with explicit bounding-box coordinates to condition action policies including Diffusion Policy (DP) and Action Chunking with Transformers (ACT). The effectiveness of the proposed FOM-SAM3 was validated on the FO-30 dataset comprising 30 physical objects across four coarse categories. Across three real-robot FO manipulation tasks, our FOM-SAM3-guided policies demonstrated robustness against distractors, discrimination ability among similar FOs, and extendibility to new FOs.