Teleoperation robots adjust help by reading operator emotions and motions
Affective Shared Autonomy: Temporal Affect Dynamics and Subjective Evaluation in Bimanual Teleoperation Tasks
RoboticsHuman-Computer Interaction
Summary
Using robots to control things remotely can be hard and frustrating for people, leading to mistakes and stress. The authors created a system that watches a person's face, heart signals, and arm movements to figure out how they're feeling while controlling a robot. When the system detects the person is upset or struggling, it gently steps in to help without taking full control. This approach helped people stay focused and feel more positive during tricky two-handed tasks. The authors also collected a new dataset with all these signals to better understand how feelings change over time during these operations.
What this means in practice
- •For robotic system designers: Create teleoperation robots that adapt assistance based on users’ emotional and physical states to improve performance and reduce frustration during complex tasks.
- •For human factors engineers: Use multimodal data fusion to monitor operator affect and enable interventions that preserve user control while increasing productive engagement.
Authors
Zhengji Liang, Guiyin Tian, Sijin Qu, Hainan Liu, Shiyan Hu
Abstract
Physical teleoperation integrates human cognitive flexibility with robotic precision, yet demanding manipulation tasks frequently induce severe cognitive workload, acute frustration, and execution breakdown. Conventional shared autonomy paradigms rely primarily on task-based rules, such as spatial error boundaries, which disregard the operator's transient affective state and risk misaligned control interventions. To address this limitation, we propose an affect-aware shared autonomy teleoperation framework that dynamically modulates robotic assistance based on real-time operator state estimation. The system estimates operator affective states from synchronized facial video, cardiac signals, and bilateral arm kinematics, outputting a seven-state affective distribution and a three-category operational abstraction (neutral, productive, adverse). Affect-aware assistance is selectively triggered when the user is detected in a continuous adverse state, preserving task-positive engagement without unnecessary disruption. The empirical user study ($N = 30$) confirms that the proposed affective assistance increases the productive states by up to 39.7% without compromising user agency. The collected dataset represents the first multimodal dataset that provides continuous visual, physiological, and operator's bilateral motion tracking of temporal affective state shifts during bimanual teleoperation. Our multimodal fusion model outperforms zero-shot baselines (Qwen, MiniCPM-V) in tracking temporal state dynamics. This real-world deployment offers a new human-centric framework that integrates visual, physiological, and motion tracking for physical human-robot interaction.