Vision language models identify key regions to read bar chart values accurately
Inside VLM Chart Reading: Tracing Value Reading from Vertical Bar Charts Across Space and Depth
Computation and Language
Summary
Some advanced AI models can answer questions about bar charts, but it wasn’t clear exactly how they figure out the exact numbers shown. The authors studied two such models and found that the top part of each bar in a chart plays a bigger role than the rest of the bar for reading values. They also discovered that these models shift where they get information from as they process the chart, moving from the legend in early stages to other parts of the input later on. This helps show how the models combine different pieces of the chart to determine precise values.
What this means in practice
- •For data visualization developers: Design better AI tools to extract exact values from bar charts by focusing on key regions identified in this study.
- •For business intelligence teams: Improve automated chart interpretation by integrating model insights about which visual features best indicate numeric values.
Authors
Tianhao Niu, Qingfu Zhu, Wanxiang Che
Abstract
Vision--language models (VLMs) can answer chart questions accurately, but output accuracy does not show how they combine the evidence needed to recover an exact value. We study vertical-bar value reading with controlled counterfactual activation patching in Qwen2.5VL-7B-Instruct and InternVL3.5-8B. The study connects three analyses: (1) The single-factor results show that the changed bar-top region restores much more answer preference than the unchanged bar body, despite containing fewer visual tokens. Legend- and series-related states also lose local recoverability earlier than bar-geometry and axis-scale states. (2) In the handoff analysis, restoration shifts from visual legend regions in early layers to prompt-series positions in middle layers. Resetting the prompt-series state selectively reduces legend-source rescue, supporting its role as a partial mediator. (3) In the factorial analysis, both models can use geometry and scale states from separate donors to favor the combined target. InternVL performs similarly when the states come from separate donors or one image, while Qwen shows lower restoration for separate donors which suggests higher context sensitivity. Together, these results provide preliminary causal evidence for localizing the internal computations that support exact bar-value reading.