Mapping computer use at multiple levels reveals user and agent differences
Comparing Computer Use Behavior Across Multiple Levels of Abstraction
Human-Computer Interaction
Summary
Comparing how people use computers is hard because computer actions are often recorded in very detailed, specific ways. The authors created a new method called MAP that groups these detailed actions into a common, simpler language at different levels, making comparison easier and clearer. They showed MAP works well by testing it on several datasets and found interesting differences: blind and sighted users act more similarly at higher task levels, while artificial agents skip helpful human behaviors like asking for help or fixing mistakes.
What this means in practice
- •For user experience teams: Identify where users struggle by comparing behavior across different interfaces and user groups using a shared action vocabulary.
- •For human-computer interaction developers: Improve assistive technology by understanding how blind and sighted users differ in navigation but converge in higher-level tasks.
- •For robotic process automation designers: Design more human-like agents by recognizing and integrating corrective and help-seeking behaviors omitted by current automated systems.$Commercial implications: Enables creating software agents with richer human-like interaction patterns, improving user acceptance and effectiveness.
Authors
Ananya Gubbi Mohanbabu, Rosiana Natalie, Anhong Guo, Amy Pavel
Abstract
Interaction traces record how people and agents use computers, and comparing them can reveal usability barriers, differences in expertise, and human--agent similarity. Yet traces are recorded as low-level, interface-specific actions, making comparisons difficult across tasks, interfaces, and input modalities. Existing approaches either compare these actions directly, reduce traces to aggregate outcomes, or summarize them at higher-level goals, obscuring where behavior differs. We present MAP (Multi-level Action Parsing), a hierarchical semantic representation and annotation pipeline that maps traces into a common vocabulary across linked levels of abstraction. MAP compares traces by sequence, recurring patterns, and effort, localizing differences by type and abstraction level. A technical evaluation across three computer use datasets shows high agreement with manual annotations and broad coverage of behavior. Applying MAP shows that sighted and blind users increasingly converge at higher levels despite navigation differences, while agents omit corrective, help-seeking, and recovery behaviors present in human traces.