Clamp improves safe embodied plans by grounding vision and actions
CLAMP: Constrained Decoding for Vision-Language Embodied Planning
Artificial Intelligence
Summary
Making robots follow instructions safely is hard because they might talk about things they don’t see or choose actions that don’t work. The authors created CLAMP, a method that uses what a robot sees and rules about actions to stop bad or impossible choices when planning. CLAMP filters out wrong next steps and favors ones that fit the scene and goals, helping plans stay realistic and safe. It learns from its own guesses to work better with new tasks and environments.
What this means in practice
- •For robotics engineers: Generate executable and safe action plans from visual inputs and instructions by integrating scene-aware constraints into language-based planners.
- •For autonomous vehicle teams: Improve decision making by excluding unsafe or infeasible actions based on current sensor observations and action models during planning.
Authors
Tianyi Ma, Parisa Kordjamshidi
Abstract
Embodied planning increasingly relies on vision-language models (VLMs) to translate instructions and visual observations into executable action sequences. However, fluent plans are not always executable. A VLM may refer to objects that are not visually observed, select actions whose required affordances are unavailable, or violate syntax and action constraints. We introduce CLAMP, a multimodal constraint-grounding framework that turns scene evidence into decoding-time constraints for a frozen VLM planner. CLAMP uses the initial observation to restrict object references to those supported by the scene, while a provided symbolic action model specifies state transitions and goals. During decoding, hard masks eliminate invalid next-token candidates, while a Hidden Markov Model (HMM)-based world-state lookahead module reweights the probabilities of the remaining feasible candidates based on action preconditions and goal reachability. This allows the planner to retain the VLM's language prior while preventing visually unsupported, unsafe, or infeasible candidates from entering the plan. For unseen tasks and environments, CLAMP adapts the HMM at test time using label-free continuations sampled from the frozen VLM. Experiments on VLABench, SafeAgentBench, and TaPA show that scene-grounded constraints improve object grounding and safety, while most remaining failures stem from perception errors or misaligned constraint specifications.