Human turns reveal detailed contributions in human agent teamwork
From Tasks to Turns: Measuring Human Labor Contributions in Human-Agent Joint Production
Human-Computer Interaction
Summary
It can be hard to tell how much each human instruction helps when people work together with AI agents to get tasks done. The authors created a way to measure the contribution of each human instruction by looking closely at the change in task quality and computer work when an instruction is given. They tested this across many interactions and found that different kinds of instructions affect outcomes and computer effort in various ways. This method helps us understand human input more clearly during collaborations with AI.
What this means in practice
- •For human-computer interaction teams: Track and analyze how individual human inputs influence task outcomes and computation in assistive agent workflows.
- •For workflow automation engineers: Improve efficiency by pinpointing which human instructions reduce computational costs while maintaining task quality in joint human-agent systems.
Authors
Likai Wang, Dezhao Su, Jing Hu, Yunyun Wan, Jingdi Lei, Xinhai Li
Abstract
In agent workflows, a brief human instruction can alter subsequent search, reasoning, and tool execution, yet final task outcomes rarely reveal the contribution of an individual instruction to human-agent joint production. We propose a Human Turn-level framework for measuring human contributions, using task quality and token usage to characterize the collaborative production outcomes associated with each human instruction. We compare continuations from adjacent checkpoints under matched models, budgets, and termination conditions, and assess stability through repeated measurements. Across four archived and simulated interaction settings, we conduct 29,124 comparisons between predicted and observed directions of quality and computational change at both Human Turn and session levels. On HumanBench, we obtain 506 valid comparisons across 85 real-user scenarios. Results show that quality and computational changes associated with human guidance depend on task conditions. Expert-style and ordinary-style guidance differ in information provision, correction, and repetition. Even when task quality after continuation is identical, remaining computational demands can differ substantially across checkpoints. Our framework advances the measurement of human labor contributions from tasks and sessions to individual turns, providing a repeatable, fine-grained foundation for studying human contributions and their allocation in human-agent joint production.