Robot system detects mismatches to avoid human interaction failures
WSM-Aware HRI: An IoT-Enhanced Framework for Early Detection and Norm-Guided Repair of Failures with LLM Guidance
RoboticsHuman-Computer Interaction
Summary
Human-robot communication can fail because robots and people often understand the world differently. The authors propose a system that uses a combination of smart sensors and language models to identify when robots and humans have different views of a situation. This system spots potential problems early by making hidden assumptions clear and checking them against the robot's sensed data. It helps prevent mistakes before they happen and explains decisions in ways people can understand.
What this means in practice
- •For robot developers: Build robots that sense and interpret human assumptions early to prevent communication failures during interaction.
- •For smart home system integrators: Integrate IoT devices with robots to detect and resolve mismatches between human instructions and robot perceptions for safer home automation.
Authors
Hanlin Zhang, Yuquan Wang, Tianwei Zhang, Zhenglong Sun
Abstract
Human-robot interaction (HRI) failures remain a major barrier to deploying robots in real-world environments. Prior work often treats failures as isolated technical faults or focuses on post-hoc recovery behaviors. In practice, many breakdowns arise because humans and robots operate under inconsistent assumptions about the current world state. We propose WSM-Aware HRI, an IoT-enhanced modular framework that unifies diverse HRI breakdowns as World-State Mismatches (WSMs) between a human's instruction-implied assumptions and a robot's grounded world model built from multimodal perception and digital augmentation. A Large Language Model (LLM) is used to make implicit assumptions explicit, map them to a small set of mismatch types, and specify the evidence needed for verification against the robot's world state. WSM-Aware HRI shifts failure handling from execution-time recovery to proactive mismatch detection during intention formation, enabling interventions guided by safety, norm compliance, and multi-user coordination with transparent explanations. We evaluate mismatch identification in ten everyday cases spanning both visual and latent-state mismatches. The system can accurately produce the expected output results, and ablations show that reliable identification depends on appropriate grounding representations and verification-oriented refinement. These results indicate that treating interaction breakdowns as explicit world-state mismatches enables earlier detection of impending failures and offers a principled mechanism for integrating external evidence and social constraints into human-robot interaction.