Vision system monitors worker attention and action for robot teamwork
A Vision Based Framework Integrating Attention and Action Cues for Interpretable Cognitive Workload Assessment in Human Robot Collaborative Assembly
Robotics
Summary
Working with robots on factory tasks can be tricky because humans need to focus and act carefully alongside machines. The authors created a camera-based method that watches where workers look and what they do, helping to measure their mental workload without needing extra sensors on the body. They tested this in a gearbox assembly task with volunteers and found it matched well with traditional workload ratings and heart activity. This system can help spot when workers hesitate or get overwhelmed, making robot helpers smarter and safer.
What this means in practice
- •For industrial automation teams: Integrate vision-based workload monitoring to adapt robot assistance during assembly tasks without body sensors.
- •For factory safety managers: Use real-time behavior cues to detect worker hesitation and high workload, improving ergonomic interventions.
Authors
Junyan Xionga, Naiyi Feng, Xingke Xia, Qihang Fan, Suchang Chen, Daqiang Guo
Abstract
The introduction of human-robot collaboration (HRC) in industrial assembly operations is revolutionizing the manufacturing landscape. In this evolving environment, operators are required to seamlessly coordinate their manual tasks with real-time task information and robotic behaviors. These demands fluctuate during operation, yet conventional workload assessments depend on body-worn physiological sensors that complicate practical deployment. Here, we present a vision-based attention--action framework for continuous and interpretable workload-related assessment in HRC assembly. The framework combines RGB-D observations with robot states and calibrated task-related areas to construct a temporally confirmed representation of operator behavior. This representation identifies where task demand is concentrated and explains how it develops when attention and action diverge, the task context changes, or the operator hesitates. We evaluated the framework in a three-level collaborative gearbox assembly experiment with ten participants, using subjective ratings and synchronized physiological signals as independent references. Raw NASA-TLX ratings confirmed increasing perceived workload across conditions, with significant effects on overall workload and its mental and temporal dimensions. The vision-derived HRC-CWL output was significantly associated with ECG-derived features in seven of nine participants with complete correlation data. Synchronized interaction episodes further showed temporal correspondence between detected hesitation and physiological activity. Real-time deployment demonstrated that the framework can operate without requiring operators to wear additional sensors. These findings support HRC-CWL as an interpretable behavioral proxy for cognitive ergonomics analysis and adaptive robot assistance, rather than a direct psychophysiological measure of workload.