Evaluation system reveals real-world robot grasp success variations
GraspBench: Execution-Aware Cross-Robot Evaluation of 6-DoF Grasp Generation in MuJoCo
RoboticsArtificial Intelligence
Summary
Grasping objects with robots is tricky because not all proposed grasps can actually be done by a real robot. The authors designed GraspBench, a testing system using simulations to check if robotic grasp plans work on different robots and objects. Their study shows that the quality of grasp proposals alone doesn't guarantee success; how the robot and object shape affect execution matters a lot. This helps better understand which robot and grasping approaches perform well in realistic situations.
What this means in practice
- •For robotics system engineers: Test and compare different grasp generation pipelines across multiple robot types using a standardized protocol to select effective grasping strategies.
- •For robotic automation integrators: Assess how robot-specific factors affect grasp success before deploying robotic grasping in industrial settings with varied object types.
Authors
Abdul Basit, Bibek Poudel, Muhammad Shafique
Abstract
Modern 6-DoF grasp generators are often evaluated using grasp scores, dataset-level metrics, or single-robot demonstrations, yet proposal quality alone does not determine whether a grasp is executable by a particular robot. We present GraspBench, a controlled MuJoCo protocol for evaluating complete grasp-generation pipelines after robot-specific inverse-kinematics filtering, motion planning, and fixed approach-close-lift execution. GraspBench evaluates six configured pipelines across four robot embodiments, five diagnostic objects, and a structured tabletop pose grid, yielding 540 scenarios per method and 3,240 method-scenario evaluations in total. Missing outputs are conservatively counted as failures, and success follows the binary lift criterion produced by each method's execution runner. GraspGen achieves the highest overall success rate (73.1%), followed by EconomicGrasp (63.5%), while robot- and object-conditioned analyses reveal substantial variation across embodiments and object geometries. GPG produces the largest native candidate sets but achieves 44.3% success, indicating that proposal volume alone does not guarantee robot-executable grasps. Cluster-aware paired intervals support the principal method differences while not estimating repeated-execution uncertainty. The results characterize complete configured pipelines on fixed scenarios rather than intrinsic model quality or stochastic repeatability, showing that grasping performance depends jointly on the generator, robot embodiment, scene, and execution protocol.