Large multimodal models show biases in evaluating charts for code generation
ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation
Computer Vision and Pattern Recognition
Summary
Building software that generates code from charts often uses large AI models to judge how good the chart outputs are. The paper’s authors created a test set called ChartJudgeBench to see how well these AI judges work. They found that the models tend to have certain biases, like favoring some chart positions, over-accepting results, struggling with style, and being too lenient after training. These issues mean current AI judges need careful checking before being used to improve chart-to-code systems.
What this means in practice
- •For software developers: Improve feedback quality when refining chart-to-code systems by validating AI judge reliability using ChartJudgeBench.
- •For user interface designers: Detect weaknesses in AI systems that assess visual chart quality to better align automatic design tools with human preferences.
Authors
Lijian Wu, Henry Hengyuan Zhao, Zijian Zhang, Jiahao Tang, Jiajun Wu, Alex Jinpeng Wang
Abstract
Building strong chart-to-code systems increasingly relies on reinforcement learning, whose effectiveness depends critically on the quality of the reward signal. Large Multimodal Models (LMMs) play a natural critical role in jointly assessing chart visual appearance and task requirements. They are therefore increasingly used as visual critics and reward models, yet their reliability as judges remains largely unexplored. To this end, we introduce ChartJudgeBench, a diagnostic vision-language benchmark for assessing LMM judges in chart-to-code workflows. It includes 1,003 Chart Perception Alignment (CPA) instances for pairwise chart comparison and 650 Chart Reasoning Judgment (CRJ) instances for binary Accept/Reject verification in Chart Reproduction and Chart Editing. Together, these tasks emulate the core judging decisions required in agentic refinement and RL-based chart optimization. Our evaluation of strong LMMs reveals four systematic limitations: (i) positional bias in pairwise comparison, (ii) a strong tendency to overpredict Accept, (iii) difficulty in matching visual styles and aesthetics, and (iv) an unexpected leniency bias in RL-trained models. These findings show that current LMM judges require explicit reliability validation before being used as critics or reward models in chart-to-code optimization. The code and data are available on ChartJudgeBench.