LLM judges struggle to match human visualization preferences

Evaluating Human-LLM Alignment on Visualization Preference

Human-Computer Interaction

Summary

Choosing the best way to show data with charts is tricky since many good options exist. The researchers looked at how well AI models agree with people when picking preferred visualizations. They found that AI tends to give very firm answers, ignoring the natural differences in human opinions. This means current AI may miss some creative or useful designs people like.

What this means in practice

Authors

Tanmay Bakshi, Chenglong Wang, Simra Shahid

Abstract

There are often multiple valid visualizations, spanning different chart types, visual encodings, and aesthetic choices, that can all support a user's goal of analyzing a dataset. Determining which of these candidate visualizations is most appropriate, however, remains challenging. Prior work has leveraged expert-curated taxonomies to guide visualization design and subsequent quality evaluation, a practical evaluation model must also account for human-centric factors. Latest visualization tools integrate Large Language Models (LLMs) to generate and refine visualizations, using LLM-as-a-judge to provide design feedback. Without a deeper understanding of whether, and in what ways, human and LLM perceptions of visualizations align, there is a real risk of deploying low-quality agents that do not reflect human judgment. We study this question through a pilot experiment - we formulate the problem as a preference task over curated chart pairs and compare the judgments of state-of-the-art vision models with those of a panel of human annotators. We establish a cautionary tale: such judges risk filtering out informative, compelling displays. We find that frontier models collapse human subjectivity, a rich signal that reveals contrasting design choices, into near-deterministic verdicts.