AI summaryⓘ
The authors explain that current methods for teaching computers to write code for charts often assume that all parts of the original data can be seen in the chart image, but this isn’t true. For example, some chart types only show summaries or proportions, not the exact raw data points. To fix this, the authors propose a new supervision method called Observation-Aligned supervision, which trains models using only the information actually visible in the chart. Their experiments show this approach helps models better match what can be observed in charts, improving performance on different datasets. This means teaching models requires careful choice of what targets to learn from, based on what is actually identifiable in the chart image.
chart-to-code generationsupervised fine-tuninglatent variablesboxplotpie charthistogramObservation-Aligned supervisionvisual language models (VLMs)code generationobservable values
Authors
Tianhao Niu, Qingfu Zhu, Wanxiang Che
Abstract
Chart-to-code generation is commonly trained with supervised fine-tuning on reference plotting scripts, implicitly treating the gold code as a fully observable target. We argue that this assumption is often invalid: many chart programs contain latent raw variables that cannot be uniquely recovered from the rendered image. For example, a boxplot exposes summary statistics rather than original samples, a pie chart reveals proportions rather than arbitrary raw values, and a histogram shows bin-level mass rather than individual observations. Supervising models to reproduce such non-identifiable quantities encourages hallucination and over-specified code generation. We introduce Observation-Aligned supervision, a rewriting framework that replaces latent raw-data targets with quantities constrained by the visual observation: box statistics for boxplots, wedge percentages for pie charts, and bin weights for histograms. Applying this framework to chart-to-code training data from two sources, we obtain the Observation-Aligned supervision target data. Experiments across multiple VLMs on ChartMimic and ChartX demonstrate consistent improvements in observable value recovery, including under both-executable evaluation. Our results suggest that improving chart-to-code models requires not only more data or advanced learning objectives or algorithms, but also supervision targets that respect what is identifiable from the chart image.