Robot task planners struggle with dementia speech but improve using past experiences
TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns
RoboticsComputation and Language
Summary
People with dementia often give unclear or confusing instructions to robots that help them, which causes the robots to make mistakes or act unsafely. The authors created TALK-Dem, a large set of real dementia-related instructions, to test how well robot planners handle these challenges. They found many robot systems perform much worse with such unclear speech but improved significantly when they used a method that recalled similar past tasks for better understanding. This highlights the need to build robot assistants that can handle imperfect human communication carefully, especially for vulnerable users.
What this means in practice
- •For robot developers: Improve assistive robot task planning to handle unclear speech patterns common in dementia patients.
- •For healthcare technology teams: Design safer home care robots by testing their understanding of real-world dementia-associated communication.
Authors
Guangxin Zhao, Yiran Hu, Yuan Cao, Chenxi Jiang, Jianfei Yang, Yegang Du, Yasuyuki Taki, Yoshifumi Kitamura, Lin Gu, Zhi Zheng
Abstract
Existing LLM-driven robot task planners rely on a taken-for-granted assumption of an ideal user whose instructions are clear, complete, and task-focused. However, when interacting with real-world users, especially those experiencing cognitive impairments, such as people living with dementia (PLWD), the planners often make mistakes and even pose physical safety risks. We proposed TALK-Dem (Talking Attributes and Linguistic Knowledge in Dementia), the first benchmark for evaluating LLM-driven robot task planning under dementia-associated verbal communication. TALK-Dem contains 4,800 instructions and covers five typical communication patterns, including Referential Imprecision, Object Substitution, Empty Speech, Topic Drift, and Intrusion, at three intensity levels. Experiments across six open-weight LLMs reveal a substantial robustness gap. Across communication patterns, open-weight models exhibited performance drops of up to 22.3 percentage points compared to ideal instructions. This revealed a critical gap and even danger for real-world applications, especially in assistive robotics, where locally deployable models are necessary due to privacy concerns and connectivity constraints. To mitigate this issue, we proposed the Context-Aware Retrieval from Experience (CARE) method, which retrieves relevant previously resolved tasks to provide task-specific interpretation and planning context. CARE generally outperformed standard prompting baselines across the six open-weight models, improving average task success by 18.1 percentage points over the vanilla prompt. These results highlighted the importance of both evaluating communication robustness and developing effective adaptation strategies for locally deployable assistive robots. The TALK-Dem dataset is publicly available at https://anonymous.4open.science/r/TALK-Dem-A6B3/.