Measuring how analysts and visualization agents work together effectively
Measuring the collaboration, not the agent
Human-Computer Interaction
Summary
When people analyze complex data, the most important knowledge about what matters often comes from experts, not just computer agents. This paper suggests that instead of judging visualization tools alone, we should measure how well humans and agents collaborate. The authors propose five ways to measure this teamwork using interaction logs, focusing on how useful the agent's suggestions are and how the human reacts. They tested their ideas with a controlled setup using simulated analysts and public manufacturing data, showing these measures can capture differences in collaboration quality.
What this means in practice
- •For data analysts: Evaluate tools that suggest visual data views by how well analysts and agents improve insights together.
- •For ui designers: Design visualization interfaces that track and use analyst feedback for better adaptive suggestions.
Tested on one dataset.
Authors
Xinxiao Li, Eisei Nakahara
Abstract
Benchmarks for visualization agents score the agent alone; the human only supplies reference answers. For high-dimensional data this misses what matters: the knowledge of which variables count is held by on-site experts, not by the model, so the useful question is how well an analyst and an agent do together. We study one specific form of collaboration, the agent proposes a view, the analyst accepts, rejects, or modifies it, and those judgments are kept for later sessions, and propose five measures for it: complementarity, intervention efficiency, appropriate reliance, knowledge retention, and agency. The first four are computed from interaction logs alone. We test the measures, not people: in a testbed for parallel-coordinates design on public manufacturing data (UCI SECOM), with a simulated analyst whose behavior we control, all four are computable and respond to the controlled differences. Whether they reveal anything about real analysts is the next step; we release the testbed for it.