Taxonomy clarifies robot responses to incomplete human instructions indoors
From Wizard-of-Oz Human-Robot Dialogue Collection to a Taxonomy of Robot Response Decisions: A Retrospective Analysis of Assistive Pilot Interactions
Robotics
Summary
Robots helping people indoors often get unclear directions because humans leave out details. The authors studied how a person pretending to be a robot responded during tasks like opening doors or feeding. From this, they created a system to categorize robot replies, like when to act, ask questions, or refuse. They tested how well humans and AI could apply these categories and explored training AI to decide by itself.
What this means in practice
- •For robot software developers: Use the taxonomy to design robot dialogue systems that better decide when to act or ask for clarification in indoor assistance tasks.
- •For ai model trainers: Train vision-language models on annotated real-world robot interactions to improve autonomous understanding of unclear instructions.
Authors
Guangping Liu, Nicholas Hawkins, Tipu Sultan, Flavio Esposito, Madi Dian
Abstract
Robots that follow natural-language instructions in everyday indoor environments must act on incomplete human utterances. Instructions often omit essential information, such as the identity of an out-of-view object, an intended destination, or the user's goal. Existing datasets contain little real-world situated dialogue and provide few practice-grounded criteria for deciding when a robot should act, confirm, clarify, or refuse. We retrospectively analyze a pilot Wizard-of-Oz study in which five participants performed everyday indoor tasks, including door opening, drawer opening, feeding, drinking, and cleaning, with a wheelchair-mounted mobile manipulator while the wizard responded without a formal communication policy. This preserved authentic user behavior but produced inconsistent robot-side decisions, motivating an explicit decision scheme. From 40 episodes, we derived a hierarchical taxonomy of six response modes (ANSWER, REPORT_DONE, REFUSE, CONFIRM, CLARIFY, ACT) and four ambiguity types (intent, referential, spatial, intelligibility). Two human annotators and an AI annotator applied the scheme to the pilot data. Clean-label rates were 91% and 89%, and Cohen's ranged from 0.72 to 0.95 across decision-point, mode, and ambiguity levels for both human-human and human-AI comparisons. Fine-tuning LLaVA-1.6-7B on taxonomy-derived labels for ACT and CLARIFY indicates the feasibility of training vision-language models using annotations from our taxonomy. Remaining boundary cases in decision-point identification and REPORT_DONE motivate a constrained protocol for more consistent dialogue collection.