Dyad dataset links human help steps in gearbox assembly tasks
DYAD: A Multimodal Dataset of Co-Located Human Assistance
Robotics
Summary
Helping robots or assistants need to understand when and how to help humans during tasks. The authors created DYAD, a detailed collection of videos and data from people working together on gearbox assembly, showing when a helper gave verbal or physical assistance. This dataset connects helpers' actions to what the task requires and requests from the person assembling. It can help improve future assistants by showing which help is needed and how people respond during real-time teamwork.
What this means in practice
- •For industrial automation teams: Develop assistive robots that understand and predict human help requests during complex assembly tasks using linked verbal and physical intervention data.$Commercial implications: Enables production of collaborative robots that provide timely and context-aware assistance on factory floors to improve efficiency and safety.
- •For augmented reality developers: Create AR systems that anticipate and support real-time task assistance by analyzing multimodal cues from workers and helpers in shared workspaces.
Authors
Akhil Ajikumar, Mahya Qorbani, Sakib Reza, Sean Andrist, Mohsen Moghaddam
Abstract
An embodied assistant working beside a person must track task state, recognize help seeking, choose how to intervene, and produce an appropriate response. Existing procedural datasets richly describe individual execution, while interactive datasets capture remote verbal instruction or undifferentiated co-working. They do not jointly link a co-located helper's verbal and physical interventions to performer requests, task state, assistance triggers, and outcomes. We introduce DYAD (DYadic Assistance Dataset), a synchronized multimodal record of human-human assistance during gearbox assembly. Across 20 sessions, one trained helper follows a guidance-first policy while assisting HoloLens 2 wearers. DYAD links 528 task-step intervals and 611 performer requests with 851 valid assistance records spanning verbal and physical help. DYAD's annotations span the assistance process; three reference tasks evaluate selected components rather than an end-to-end system: causal step understanding, pre-onset mode anticipation, and instructor response generation. On 829 eligible mode events, the strongest four-seed RGB mean is 0.548 +/- 0.007 macro-F1; causal metadata reaches 0.624 and a privileged trigger mapping 0.915, revealing information not recovered from pre-onset RGB. DYAD's contribution is not scale, but a linked interaction structure spanning help seeking, intervention choice, execution, and outcome under egocentric and workspace sensing.