Chart layer understanding and editing reveal visibility challenges

Charts Are Beyond Pixels: Probing for Layer-Wise Chart Understanding and Editing

Computer Vision and Pattern RecognitionMachine Learning

Summary

Charts are made of different layers that have specific roles and meanings, like pieces of a puzzle stacked in order. The authors created a new benchmark called LayerWiseBench to see if computer models can understand and edit charts by looking at these layers one by one, instead of just the final image. They found that models do well recognizing the parts and their roles but often struggle with figuring out which parts are in front or behind others. This work highlights how tricky it is for computers to grasp the overlapping and visibility between chart components.

What this means in practice

Authors

Xiaochuan Zhong, Yifan Hou, Chenxi Pang, Shaobo Cui

Abstract

Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This structural view motivates evaluating whether models can understand and manipulate charts at the layer level. Existing chart benchmarks, however, primarily assess the correctness or fidelity of final outputs and do not directly evaluate these layer-wise behaviors. We present LayerWiseBench, a benchmark organized around three core concepts, layer attribution, layer binding, and visibility ordering, that structure its chart-understanding and chart-editing evaluations. Generated from executable chart programs, LayerWiseBench pairs each rendered chart with spatially aligned per-layer RGBA assets and construction-derived labels for functional roles, semantic bindings, and visibility relations. From this layer-wise representation, we derive controlled understanding questions, editing targets, reference images, and evaluation regions. It contains 2,800 source charts across 14 chart paradigms, from which we derive 7,329 layer-wise understanding questions and 53,791 instruction-guided editing variants. Among the evaluated VLMs, Qwen3.5-27B, which achieves the highest QA macro-average, obtains 93.04% accuracy on layer attribution and 97.46% on layer binding, but only 61.46% on visibility ordering. Across the four evaluated image editors, overall mIoU ranges from 1.49% to 4.93%, and visibility-constrained edits have the lowest mIoU for every editor, ranging from 0.37% to 2.00%. Taken together, these results identify tasks involving front-to-back relations between overlapping components as a recurring challenge across understanding and editing, motivating more explicit modeling of component identity and visibility relations.