MIGU improves robot understanding of uncertain speech and gestures
MIGU: Multimodal Instruction Grounding under Uncertainty for Manipulation Planning
Robotics
Summary
Robots need to understand people’s spoken instructions and gestures to work well around humans. The authors created MIGU, a system that combines robot vision and language understanding to figure out what a person means, even when their gestures or words are unclear. MIGU assesses uncertainty in where a person is pointing and what they say, then uses that info to decide if the robot should ask for clarification or take action. It helps robots better understand instructions in three-dimensional spaces, making them more reliable helpers.
What this means in practice
- •For robotics engineers: Enable robots to interpret uncertain human speech and pointing gestures more reliably for precise task execution.
- •For industrial automation teams: Improve factory robots' ability to understand ambiguous operator instructions combining gestures and verbal commands.
Authors
Mingke Lu, Anxing Xiao, David Hsu
Abstract
Understanding natural human instructions is crucial for deploying robots in human-centric environments. We study multimodal instruction grounding, where language and gesture provide complementary but uncertain cues. We present MIGU, a modular framework that combines semantic and geometric evidence into a unified grounding belief and connects it to manipulation planning. MIGU constructs a 3D geometric likelihood by propagating viewing-direction and depth uncertainty through eye-finger geometry while accounting for hand-direction estimation error. A vision-language model (VLM) provides semantic priors over candidate objects and regions, which are combined with the geometric likelihood through Bayes-inspired fusion. The resulting belief supports behavior planning to either proceed directly to downstream planning or request clarification. Grounded targets then define goals for mobile manipulation and tabletop task-and-motion planning. On a real-world benchmark, MIGU outperforms all evaluated baselines, while ablations support the benefit of explicit multimodal uncertainty modeling. Project website: multimodal-instruction.github.io