Inducing Task Models from Computer-Use Traces
2026-08-20 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors present a method called Task Model Induction (TMI) that can figure out what different tasks people are doing on a computer just by looking at low-level activity, like mouse clicks and keystrokes. Unlike past methods, TMI can separate tasks that happen at the same time and build detailed models showing the goals and steps of each task. They tested TMI on controlled data and found it can accurately identify overlapping tasks and recreate most of the actions performed. Also, the skills learned from these models help improve task recognition by a significant margin.
Task modelingWorkflow inductionRecursive goal decompositionControl flowNaturalistic computer useMulti-threaded activityTask recognitionHuman-computer interactionActivity tracingMachine learning
Authors
Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang
Abstract
Naturalistic computer-use traces, passively recorded screenshots and mouse or keyboard actions, are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done. Such models matter as computer-use agents enter real work, where agents need to learn how tasks are actually performed, and organizations need to audit and reuse that knowledge. However, inducing such task models is challenging, as activity is observed only as low-level events and real-world work is multi-threaded with interleaved goals. Existing methods assume a given task or a single workflow, and produce step-level summaries rather than structured task models. We introduce Task Model Induction (TMI), which (i) discovers the latent tasks in an unconstrained trace, disentangling concurrent activity, and (ii) for each latent task, induces a task model pairing a hierarchical objective model of recursive goal decomposition with a procedure model of the control flow that organized the execution. Intrinsically, on controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings and reconstructs 74.9% of the observed execution steps, far more than the strongest workflow induction baseline. Extrinsically, skills derived from TMI's task models improve held-out task accuracy by 30.0% over the strongest baseline.