Imagination guided robot planning improves object finding and manipulation
Imagine-TAMP: Imagination-Guided Task and Motion Planning in Partial Observability
Robotics
Summary
Finding and moving objects in messy spaces is tough for robots when they can't see everything clearly. The authors created Imagine-TAMP, a system where the robot imagines different scenes and plans before it moves, helping it decide when to look again or move objects blocking its view. It uses language-based hints and guesses about hidden parts of the scene to make smarter plans. This method helps robots find targets faster and with less wasted effort, even when they can't see the whole area at once.
What this means in practice
- •For robotics engineers: Enable robots to efficiently plan object search and manipulation in cluttered, partially hidden environments by imagining scenes before acting.
- •For warehouse automation teams: Improve robot efficiency in locating and handling occluded items on shelves by using imagination-guided task and motion planning.
Authors
Antareep Singha, Shivaram Kumar, Yoonwoo Kim, Yoonchang Sung
Abstract
Robots operating in cluttered environments must often manipulate objects whose locations are only partially observable. A central challenge is deciding whether to acquire another observation or to first manipulate objects that may occlude the target. Conventional task and motion planning (TAMP) approaches typically make this decision using symbolic action costs or expensive geometric planning, neither of which adequately captures how likely an observation is to reveal an occluded target. We introduce Imagine-TAMP, an interleaved planning and execution framework that uses semantic and geometric imagination to compare alternative task-level strategies under partial observability before committing to expensive motion planning. A vision-language model shapes a particle belief over target locations using commonsense relationships between the target and visible objects, while a generative scene model estimates plausible geometry in unobserved regions. Given a target hypothesis and imagined scene, Imagine-TAMP generates multiple symbolic plan skeletons and assigns non-unit costs that approximate both manipulation effort and target visibility from sensing actions, distinguishing a short but poorly informative observation strategy from a longer strategy that first manipulates an occluder to better expose the target. The selected skeleton is then refined into a feasible continuous plan and executed, with new observations updating the belief and triggering replanning when necessary. Experiments show that imagination-guided evaluation improves observation-versus-manipulation decisions: in viewpoint-constrained shelf scenes, non-unit geometric evaluation increases success from 46.0% to 84.0%, while semantic belief shaping further reduces manipulation and replanning. On a real robot, the complete system reduces planning time by 32% relative to a geometry-only ablation.