Visual models improve robot planning under uncertainty indoors
Bridging Learned Visual Perception and Symbolic Belief-Space Planning
Artificial IntelligenceRobotics
Summary
Robots often must make decisions without fully knowing what's around them. Previous methods used language-vision models either to plan actions directly from images or to convert images into clear facts, but both ignored uncertainty in what the robot sees. The authors show a new way to handle uncertainty by representing what the robot might perceive as a set of possible worlds with probabilities. This lets the robot plan better and more reliably in uncertain situations, as shown in home robot simulations.
What this means in practice
- •For household robot developers: Create robots that can make safer and more reliable plans despite uncertain or incomplete perceptions of home environments.
- •For industrial automation engineers: Improve robots in factories or warehouses that must act with partial knowledge by planning over possible states inferred from visual data.
Tested on simulated data.
Authors
Guy Azran, Michael Navat, Sarah Keren
Abstract
In partially observable settings, agents must act without full knowledge of the world state and rely on uncertain state-estimation pipelines. Obtaining grounded and verifiable symbolic plans under such uncertainty remains a key challenge. Recent work has integrated Vision-Language Models (VLMs) to bridge perception and symbolic reasoning, following two main paradigms. The first, VLM-as-planner, maps images directly to action sequences, and the second, VLM-as-grounder, grounds observations into symbolic predicates used as the initial state by off-the-shelf planners. Both approaches ignore uncertainty in the planning process, compromising robustness. We introduce a third paradigm, VLM-as-probabilistic-grounder, a novel approach that captures the uncertainty of VLM predicate groundings as a probability distribution over symbolic states. This enables planning in belief space and producing robust plans under uncertainty. Experiments in simulated household robot settings show improved robustness and task success over deterministic grounding, underscoring how our approach leverages foundation models for reliable planning under uncertainty.