RoboSeg: Online Part-Level Semantic Reconstruction for Robotic Manipulation via a Single Eye-in-Hand Camera
2026-08-10 • Robotics
Robotics
AI summaryⓘ
The authors developed RoboSeg, a system that helps robots recognize and handle specific object parts like handles or triggers, instead of just the whole objects. RoboSeg combines a vision-language model with real-time 3D scanning to build a detailed map showing these parts without needing pre-made 3D models. It then uses this map to plan where to grasp based on the part needed for a task. Tests showed RoboSeg can accurately identify parts and select the right grasp points for successful object manipulation.
robotic manipulationvision-language modelRGB-D scanningsemantic segmentationtruncated signed distance function (TSDF)AnyGrasp6-DoF graspingpart-level segmentationsemantic reconstruction
Authors
Zhaochen Lan, Mengxiang Lin
Abstract
Robotic manipulation requires perception systemsthat identify actionable parts such as handles, rims, triggers,and tool tips, not merely object categories or point clouds. This paper presents RoboSeg, a part-level semantic reconstructionsystem that links vision-language model (VLM) functional-partdiscovery, asynchronous online RGB-D semantic reconstruc-tion, and task-oriented grasp generation without requiring CAD models or pre-scanned meshes. RoboSeg queries a VLM onthe initial RGB observation to obtain compact functional part prompts, then scans with two asynchronous streams: a high-frequency geometry thread for RGB-D odometry and truncated signed distance function (TSDF) fusion, and a keyframe-triggered semantic thread for SAM3 part masks. Projectedmasks are fused by voxel-level temporal voting into a persistentpart-labeled point cloud; RoboSeg uses this map to assign AnyGrasp 6-DoF candidates to semantic parts and select grasps consistent with the task-relevant part label. RoboSeg reaches 83.4% mean part intersection-over-union (mIoU) over manually labeled objects; in a 24-trial physical pilot across fourobjects and eight tasks, the selected grasp contacts the requestedpart in all trials and achieves 21/24 combined task successes.These results characterize RoboSeg as a semantic indexing layerfor task-conditioned manipulation, with AnyGrasp retained asthe proposal generator.