Vision language planning improves by blocking impossible actions with context
CLAMP: Constrained Decoding for Vision-Language Embodied Planning
Artificial Intelligence
Summary
Sometimes, robots or AI plans use language and vision to decide what to do, but they can pick actions that don't match what they see or can't actually do. The authors created CLAMP, a system that uses what is visible and some rules about actions to prevent the AI from choosing impossible or unsafe steps. CLAMP works by blocking wrong choices during planning while still letting the AI use its language skills. In tests, this approach made the plans safer and more accurate, though some mistakes still happened due to errors in seeing or understanding the scene.
Vision-language modelsEmbodied planningConstrained decodingAction affordancesSymbolic action modelHidden Markov ModelObject groundingSafety in AICommand executionScene understanding
Authors
Tianyi Ma, Parisa Kordjamshidi
Abstract
Embodied planning increasingly relies on vision-language models (VLMs) to translate instructions and visual observations into executable action sequences. However, fluent plans are not always executable. A VLM may refer to objects that are not visually observed, select actions whose required affordances are unavailable, or violate syntax and action constraints. We introduce CLAMP, a multimodal constraint-grounding framework that turns scene evidence into decoding-time constraints for a frozen VLM planner. CLAMP uses the initial observation to restrict object references to those supported by the scene, while a provided symbolic action model specifies state transitions and goals. During decoding, hard masks eliminate invalid next-token candidates, while a Hidden Markov Model (HMM)-based world-state lookahead module reweights the probabilities of the remaining feasible candidates based on action preconditions and goal reachability. This allows the planner to retain the VLM's language prior while preventing visually unsupported, unsafe, or infeasible candidates from entering the plan. For unseen tasks and environments, CLAMP adapts the HMM at test time using label-free continuations sampled from the frozen VLM. Experiments on VLABench, SafeAgentBench, and TaPA show that scene-grounded constraints improve object grounding and safety, while most remaining failures stem from perception errors or misaligned constraint specifications.