Multi modal grasping system improves robot picking in cluttered scenes
MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes
RoboticsComputer Vision and Pattern Recognition
Summary
Robots often need to grab objects from messy piles, which is hard because sensors can miss details or produce blurry views. The authors created MMGNet, a system that uses both 3D shape data and color images together to better understand objects and decide how to grasp them. MMGNet predicts the best way to approach and pick up items using combined visual clues, resulting in more successful picks in tests with real and simulated cluttered scenes. This approach improves robotic grasping by integrating different types of information smartly.
What this means in practice
- •For robotic system integrators: Implement robotic arms that pick objects from cluttered environments more reliably with combined 3D and image data.
- •For warehouse automation engineers: Improve automated picking solutions that handle varied and piled items by integrating multi-modal grasp synthesis.
Authors
Rui Cao, Jia-Hui Pan, Yichuan Li, Yunfan Yu, Chi-Wing Fu, Jiangliu Wang, Yun-Hui Liu
Abstract
Grasping is a foundational capability in robotic manipulation. Scene-level 6-DoF grasp synthesis relies on point clouds for metric geometry, yet real-world point observations are often noisy, incomplete, and over-smoothed. RGB images provide object-region, boundary, and appearance cues, yet two bottlenecks limit their effective use in scene representation and local grasp reasoning. We present MMGNet, a Multi-Modal Grasp Synthesis Network that addresses these bottlenecks through hierarchical image-to-3D feature injection and rotational graspness modeling. Hierarchical injection integrates multi-scale image features throughout 3D representation formation, while the rotational graspness jointly predicts the grasp approach direction and in-plane rotation before organizing local image-informed features through grasp-shaded grouping. Ablation studies show that these two modules provide complementary gains. MMGNet achieves a mean AP of 60.34 on GraspNet-1Billion with RealSense, surpassing all compared methods. In real-world experiments with adversarial grasp objects, MMGNet improves the grasp success rate over AnyGrasp by 8.46 percentage points. \emph{Code, models, and data will be released upon publication.}