Large language models struggle to analyze complex multi-chart scientific figures
Do you "see" what I "see"? A Multi-panel Visual Question Answer Dataset for Large Language Model Chart Analysis
Digital LibrariesArtificial IntelligenceInformation Retrieval
Summary
Scientific papers often include charts that hold important data, but computers find it hard to understand these pictures. The authors tested how well large language and multimodal models (LMMs) can answer questions about these charts, especially when there are many charts or details in one figure. They found that as charts get more complex, these models make more mistakes. They also shared the code and data they used to help others study this problem.
What this means in practice
- •For scientific publisher platforms: Automatically extract and analyze data trends from complex scientific figures to enhance searchability and accessibility in digital libraries.
- •For data visualization software developers: Incorporate evaluation benchmarks for multi-panel chart understanding to improve model interpretation features in visualization tools.
Authors
Daniel J. Evans, J. P. Naiman, JooYoung Seo, J. Stephen Downie
Abstract
Digital libraries increasingly host scientific publications whose figures encode primary research findings, yet making this visual content searchable, accessible, and reusable remains an open challenge. Large language and multimodal models (LLMs/LMMs) provide the possibility to automate many figure parsing tasks such as determining data trends or extracting data from plots. In this work, we consider if these models can reliably support task-oriented visual data analysis when presented with synthetic, but realistic, scientific figures that contain multiple charts. Our Visual Question Answering (VQA) process scaffolds our tasks along a four-level description model from basic conceptual answers to interpretive and analytical understanding. Additionally, we outline our process for synthetically generating our single and multi-panel (charts in which there are multiple panels present) plots and testing the data with various LMMs using quantitative analysis. In general, we find LMMs become less accurate with increasing figure complexity as the number of panels, number of plot elements, and difficulty of questions increase. Finally, because LMM accuracy is expected to change rapidly, our analysis also considers cost relative to performance as a metric to inform future user decisions. We provide the detailed question-answer pairs used in this analysis, along with the code used to generate random plots, question-answer pairs and detailed bounding-box metadata for chart elements. The code and data are available at https://github.com/ReadingTimeMachine/LLM_VQA_MultiPanel and https://huggingface.co/datasets/ReadingTimeMachine/visual\_qa\_multipanel