Bridging Semantics and Physics with Constrained LLMs for Safe and Trustworthy Robotic Manipulation
Robotics
Summary
The authors developed a system to help robots follow language instructions safely in messy kitchens. They use a special method to check if the robot's planned moves can actually happen without bumping into things. Their system understands objects from camera data and only lets safe, checked movements reach the robot. Tests show their approach works well for pouring and grasping tasks, doing better than a simple pre-set method on harder tasks. This helps close the gap between what a robot understands from language and what it can physically do.
Large Language ModelsRobot Motion PlanningCollision AvoidanceRGB-D SensingKinematic ConstraintsTask VerificationMoveIt Task ConstructorLanguage-Action GapModel Context ProtocolRobot Manipulation
Authors
Wenhao Hong, Lan Wei, Dandan Zhang
Abstract
A language-guided robot operating in a real kitchen must do more than produce a plan that appears correct. It must also execute that plan safely in cluttered environments under imperfect perception. Large language models (LLM) can decompose instructions into action sequences, yet a language-action gap remains: a plan may appear valid linguistically while being physically infeasible under kinematic and collision constraints. We bridge this gap by formalizing the reasoning-execution boundary as a typed contract. From RGB-D observations, the system grounds perceived objects in an explicit, collision-aware scene model and constrains language-level decisions through schema-validated tool calls defined by the Model Context Protocol (MCP), rejecting malformed commands before they reach the robot. Each validated call is deterministically grounded in a MoveIt Task Constructor pipeline, where candidate motions are evaluated against the reconstructed planning scene in a verify-then-act step. Only trajectories that pass both kinematic and collision checks are sent to the robot. On a physical UFactory 850, the method achieves up to 80% success across ten trials per task on pouring tasks involving liquids, granular media, and discrete solids. It achieves 90% success on a grasp-and-place task using the same planning, protocol, and verification stack. Although a scripted policy slightly outperforms our method on the easiest task, its success rate falls to 10% on the hardest, compared with 60% for our method.