Chart models struggle with overlapping parts in layer editing tasks
Charts Are Beyond Pixels: Probing for Layer-Wise Chart Understanding and Editing
Computer Vision and Pattern RecognitionMachine Learning
Summary
Charts are made of different layers, each with parts that have specific roles and relationships. The authors created a new test called LayerWiseBench to see how well computer models understand and edit these chart layers individually, rather than just the final picture. They found that while some models are very good at identifying and matching parts in the right layers, all models have trouble with understanding which parts appear in front or behind others. Editing tools also struggle most with changes that depend on these front-back relationships, highlighting a need for better ways to handle the order and visibility of chart elements.
ChartsLayer attributionLayer bindingVisibility orderingVisual language modelsRGBA layersChart editingBenchmarkImage segmentationOverlapping components
Authors
Xiaochuan Zhong, Yifan Hou, Chenxi Pang, Shaobo Cui
Abstract
Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This structural view motivates evaluating whether models can understand and manipulate charts at the layer level. Existing chart benchmarks, however, primarily assess the correctness or fidelity of final outputs and do not directly evaluate these layer-wise behaviors. We present LayerWiseBench, a benchmark organized around three core concepts, layer attribution, layer binding, and visibility ordering, that structure its chart-understanding and chart-editing evaluations. Generated from executable chart programs, LayerWiseBench pairs each rendered chart with spatially aligned per-layer RGBA assets and construction-derived labels for functional roles, semantic bindings, and visibility relations. From this layer-wise representation, we derive controlled understanding questions, editing targets, reference images, and evaluation regions. It contains 2,800 source charts across 14 chart paradigms, from which we derive 7,329 layer-wise understanding questions and 53,791 instruction-guided editing variants. Among the evaluated VLMs, Qwen3.5-27B, which achieves the highest QA macro-average, obtains 93.04% accuracy on layer attribution and 97.46% on layer binding, but only 61.46% on visibility ordering. Across the four evaluated image editors, overall mIoU ranges from 1.49% to 4.93%, and visibility-constrained edits have the lowest mIoU for every editor, ranging from 0.37% to 2.00%. Taken together, these results identify tasks involving front-to-back relations between overlapping components as a recurring challenge across understanding and editing, motivating more explicit modeling of component identity and visibility relations.