LLM judges struggle to match human visualization preferences
Evaluating Human-LLM Alignment on Visualization Preference
Human-Computer Interaction
Summary
Choosing the best way to show data with charts is tricky since many good options exist. The researchers looked at how well AI models agree with people when picking preferred visualizations. They found that AI tends to give very firm answers, ignoring the natural differences in human opinions. This means current AI may miss some creative or useful designs people like.
What this means in practice
- •For data visualization developers: Improve visualization tools by accounting for human subjective preferences instead of relying solely on AI judgments.
- •For business intelligence teams: Avoid overreliance on automated visualization recommendations that do not reflect diverse user perspectives when analyzing data.
Authors
Tanmay Bakshi, Chenglong Wang, Simra Shahid
Abstract
There are often multiple valid visualizations, spanning different chart types, visual encodings, and aesthetic choices, that can all support a user's goal of analyzing a dataset. Determining which of these candidate visualizations is most appropriate, however, remains challenging. Prior work has leveraged expert-curated taxonomies to guide visualization design and subsequent quality evaluation, a practical evaluation model must also account for human-centric factors. Latest visualization tools integrate Large Language Models (LLMs) to generate and refine visualizations, using LLM-as-a-judge to provide design feedback. Without a deeper understanding of whether, and in what ways, human and LLM perceptions of visualizations align, there is a real risk of deploying low-quality agents that do not reflect human judgment. We study this question through a pilot experiment - we formulate the problem as a preference task over curated chart pairs and compare the judgments of state-of-the-art vision models with those of a panel of human annotators. We establish a cautionary tale: such judges risk filtering out informative, compelling displays. We find that frontier models collapse human subjectivity, a rich signal that reveals contrasting design choices, into near-deterministic verdicts.