Robot manipulation policies struggle with everyday common sense tasks
MolmoSpaces-CommonSense: Evaluating Commonsense in Robotic Manipulation Policies
Robotics
Summary
Robots often need to understand messy and cluttered environments to perform tasks correctly, but current tests usually use clean and simple settings. The authors created a new set of tasks that require robots to use common sense, like knowing which objects are hidden or how to pick things up properly. They found that many existing robot systems make mistakes often in these tougher, more realistic scenarios. This shows that robots aren't yet ready to handle real-world complexity as well as we might hope.
What this means in practice
- •For robotics engineers: Evaluate and improve robot control policies in complex, cluttered scenes requiring common sense reasoning.
- •For robotics perception teams: Develop better visual and reasoning components to detect occluded and hidden objects in manipulation tasks.
Authors
Arjun Guru, Max Argus, Jae Sung Park, Jiafei Duan, Ali Farhadi, Ranjay Krishna
Abstract
Real-world robotic manipulation requires reasoning about cluttered scenes, partially observable or hidden objects, and semantically appropriate actions, in addition to low-level execution within those scenes. However, existing simulation benchmarks primarily test the latter and remain dominated by comparatively clean environments with limited clutter and narrowly scoped task structure, leaving many forms of commonsense reasoning untested. We address this by introducing a taxonomy of reasoning processes for robotic manipulation grounded in developmental cognitive science. We operationalize this taxonomy through a new benchmark of robot tasks, MolmoSpaces-CommonSense, that requires reasoning about occlusion, containment, support relations, semantically correct grasp points, spatial affordances, and hidden objects. These evaluations require policies not only to recognize relevant scene structure but also to execute correct actions in them. We evaluate several zero-shot generalist manipulation policies. Across model types and architectures, we observe consistent failures in commonsense reasoning: success rates in cluttered scenes drop by over 50% as distractors increase, policies act on incorrect objects in cluttered settings 30% of the time, and affordance-sensitive grasping remains unreliable even with highly specific language prompts, with correct object part selection happening only about 50% of the time. Further, VLM-based success estimation struggles to reliably evaluate rollouts in these settings, suggesting that the challenge extends beyond control to the broader perception-reasoning-action stack. Our results suggest that existing benchmarks may substantially overestimate zero-shot real-world readiness. Code and policy leaderboard link at https://github.com/allenai/molmospaces.