Foundation model powered robot actively disambiguates tasks with closed loop feedback
Actively Resolving Contextual Uncertainty for Underspecified Tasks in Natural Language
RoboticsArtificial Intelligence
Summary
Robots often get confused when people give unclear instructions or when they don’t know the environment well. The authors created a system called CLUE that helps robots figure out what they need to do by asking questions and exploring their surroundings step by step. CLUE uses a language model to guess what might be important for the task and then checks those guesses by moving around and looking. This approach works much better than previous ones, getting near the success rate of an ideal system that already knows everything.
What this means in practice
- •For robotics developers: Enable robots to interactively clarify unclear natural language tasks and environments to successfully complete complex real-world tasks.
- •For warehouse automation teams: Improve robot handling of vague instructions and dynamic environments, reducing failures from missing or ambiguous task details.
Authors
Zachary Ravichandran, Jonathan Diller, Fernando Cladera, Varun Murali, George J. Pappas, Vijay Kumar
Abstract
Foundation models provide robots with the ability to interpret natural language and reason about environmental context, yet most language-conditioned policies assume that goals are well-specified and that task-relevant information is provided upfront via a prior map. Operating in unfamiliar environments with underspecified tasks entails high contextual uncertainty: the robot must jointly infer what constitutes task success, what constitutes relevant information, and where (or whether) that information exists. We address these limitations via CLUE (Closed-Loop contextual Uncertainty rEsolution), a framework for actively resolving contextual uncertainty given underspecified tasks in natural language. CLUE uses an LLM-derived policy to hypothesize task-relevant concepts and potential plans. It then uses a language-embedded map, which is constructed online, to ground these hypotheses into actions. The policy sequentially evaluates hypotheses via closed-loop environment interaction and refines its plans as it gathers new information. We deploy CLUE on a Boston Dynamics Spot across three real indoor and outdoor environments spanning 15 tasks that require object disambiguation, functional inference, and occlusion reasoning. CLUE achieves a success rate within 7 percentage points of an oracle policy and outperforms an LLM-enabled planner without closed-loop feedback by a 4x margin. Supporting experiments demonstrate that simply building and then querying a language-enriched map is insufficient to resolve complex contextual planning tasks; these approaches achieve roughly one third the success rate of CLUE while requiring over 10x more VLM tokens. We provide additional information at https://zacravichandran.github.io/CLUE.