Vision language models improve misleading chart question answering with DAG supervision
VizDAGuide: Misleading Chart Question Answering in Compact Vision Language Models via DAG-Structured Supervision
Computer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
Charts can be tricky and sometimes misleading, causing AI models that read them to give wrong answers. The authors created a new training method called VizDAGuide that helps these models think step-by-step by following connected reasoning steps laid out in a directed graph. This approach taught models to better understand and answer questions about misleading charts using one smart training target, improving performance over older methods. They tested it with different models and question types, showing better accuracy and more reliable reasoning.
What this means in practice
- •For data visualization tool developers: Improve AI assistants that explain or verify charts by training them to handle misleading visual data more accurately.
- •For business intelligence teams: Enhance automated systems that generate answers from business charts to reduce errors from misleading graph interpretations.
Authors
Haiyang Zhang, Sicheng Song, Zixin Chen, Jie Gu, Huiya Ou, Chenhui Li, Changbo Wang
Abstract
Misleading charts can lead vision-language models to conclusions that conflict with underlying data. Existing work focuses on inference-time strategies, while training-time supervision remains less explored. We present VizDAGuide, a DAG-structured supervision method for misleading-chart question answering. The method represents intermediate reasoning states and their dependencies as a directed acyclic graph, assigns supervision weights according to their roles, and serializes the graph into a unified training target for open-ended and multiple-choice QA with single-pass inference. Experiments on two benchmarks across Qwen3-VL and InternVL3.5 models show improvements over a content- and weight-matched Linear CoT-SFT baseline on both answer formats. Process-level evaluation shows gains in grounding and operation execution, while ablations indicate that dependency structure and weighted supervision are most effective when combined. These results support DAG-structured weighted supervision for misleading-chart question answering.