LegendBench diagnoses chart legend reading errors in AI models
LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventions
Computer Vision and Pattern Recognition
Summary
Charts have legends that explain what the colors or symbols mean, but it's hard for AI models to correctly connect those legends to the right parts of the chart. The authors created LegendBench, a tool that makes test charts with tricky legends to see where AI models make mistakes. This helps find specific problems with how the AI reads legends, and by fixing those, models get better at understanding charts overall. They also tested different tweaks to improve AI performance and analyzed how models handle tricky legend details.
What this means in practice
- •For data visualization engineers: Assess AI tools to improve their accuracy in matching chart legends to visual data marks during automated chart analysis.
- •For business intelligence teams: Improve reliability of automated chart interpretation in dashboards by identifying and correcting legend-related AI errors.
Authors
Xinnuo Zhang, Zhike Tang, Jing Xu, Haoyuan Zhao, Weikai Yang
Abstract
Legends are fundamental to chart understanding, as reliable interpretation requires correctly binding legend entries to corresponding visual marks. While vision-language models (VLMs) are increasingly applied to chart understanding, their legend understanding is poorly diagnosed by aggregate accuracy, which can be satisfied by superficial shortcuts and confound legend-specific errors with other reasoning failures. To enable fine-grained diagnosis and controlled testing, we introduce LegendBench, a parametric benchmark and generation pipeline that produces targeted legend-centric test cases. LegendBench contributes (1) a capability-task taxonomy spanning legend parsing, legend grounding, legend-conditioned reasoning, and legend-aware abstention to localize failures, and (2) counterfactual group generation, where each base chart yields multiple variants under controlled legend interventions to probe model invariance and sensitivity. Using LegendBench, we evaluate both general-purpose VLMs and specialized chart models and generate their capability profiles, revealing persistent bottlenecks in reliable legend-to-mark binding and counterfactual consistency. We then use these capability profiles to guide targeted fine-tuning, demonstrating that bottleneck-specific interventions can effectively close the localized capability gaps and generalize to unseen data. We further leverage our counterfactual design to conduct fine-grained diagnostic experiments, analyzing encoding-channel effects, legend-order shortcuts, and abstention under varying visibility.