Benchmark enables better evaluation of diverse scientific diagrams
SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams
Computation and Language
Summary
Scientific diagrams come in many different types, each with unique features that experts consider when judging quality. The authors identify that current automatic tests do not separate these diagram types or match human preferences well. They created SciDiagBench, a dataset that groups scientific diagrams into 11 types and uses human preferences to guide evaluation criteria. This work helps computers better judge scientific diagrams in ways that align with how people do.
What this means in practice
- •For ai developers: Improve automated evaluation tools for scientific diagram generation to better reflect expert judgments based on diagram type.
- •For scientific publishers: Support the peer review process by providing automated assessments of scientific diagrams that align more closely with human preferences.
Authors
Hao Wang, Zixuan Li, Long Bai, Xiaolong Jin, Jiafeng Guo, Tat-Seng Chua, Tian Tian
Abstract
Recent agents have demonstrated strong capabilities in generating images from textual prompts, motivating their utilization for scientific diagram generation. Scientific diagrams naturally span diverse types, each characterized by distinct visual structures and content organization. Consequently, during peer review and other scientific evaluation, human experts naturally consider type-specific criteria when assessing their quality. However, existing VLM-based evaluation benchmarks fail to reflect the above human evaluation practices for two main reasons: 1) benchmark datasets typically lack a systematic categorization of different diagram types and thus adopt the same generic criteria for all diagrams, and 2) their evaluation criteria are often designed heuristically, without explicit alignment to human evaluation preferences. Motivated by these limitations, we introduce SciDiagBench, which consists of a benchmark dataset that systematically categorizes scientific diagrams into 11 types and a human preference-guided evaluation method. Specifically, SciDiagBench contains 165 scientific diagrams across 10 broad domains, with types including graphical abstracts, technical roadmaps, and illustrative cases. Based on this dataset, SciDiagBench develops an evaluation method that leverages a preference-guided calibration process to incorporate human preferences into the generation of type-specific criteria. Ultimately, SciDiagBench enables more reliable automated evaluation that better aligns with human evaluation across diverse scientific diagrams.