Chart reasoning agents provide verifiable stepwise reasoning for audit
Monitorable Chart Reasoning Agents via Verifiable Process Rewards
Computer Vision and Pattern Recognition
Summary
Charts show information that people want to understand and use, but it can be hard to trust how AI tools get their answers from charts. The authors created Chart-RVR, which breaks down chart reading into clear steps: first figuring out the chart type, then extracting the data, and finally showing the detailed calculations to reach the answer. This way, anyone can check if the AI made a mistake at any step rather than just seeing the final answer. Their tests show it not only performs well but also produces explanations that can be trusted more than previous methods.
What this means in practice
- •For business intelligence teams: Verify and audit automated insights from charts by tracing through stepwise reasoning, improving trust in AI-generated business reports.
- •For financial audit departments: Check the AI's intermediate steps when interpreting financial charts to catch misreadings or errors before decisions are made.
Authors
Sanchit Sinha, Oana Frunza, Kashif Rasul, Aidong Zhang
Abstract
Chart reasoning agents are increasingly used to extract actionable insights in critical domains, achieving state-of-the-art performance on multiple benchmarks. Yet, high benchmark accuracy alone is insufficient for deployment, where stakeholders must be able to audit and verify how a model reaches its answer. Existing LVLM-based chart agents produce either answer-only predictions or free-form rationales that are hard to verify, obscuring whether an error arose from misreading the chart, extracting a wrong value, or miscomputing. We propose Chart-RVR, a reinforcement learning framework for training monitorable chart agents with verifiable process rewards. Chart-RVR decomposes chart reasoning into three auditable blocks: Structure, identifying the chart type; Evidence, reconstructing the underlying data table in JSON; and Derivation, exposing the stepwise trace that computes the answer. Across six in-domain and out-of-domain benchmarks, Chart-RVR attains state-of-the-art accuracy among comparable-sized LVLMs. Beyond accuracy, we assess monitorability using a triangulated protocol that combines ground-truth surrogate metrics, an oracle information-gain measure, and an LLM-as-auditor scoring Process Verifiability and Evidence Localization, showing that Chart-RVR yields rationales that are markedly more verifiable and evidence-grounded than those from CoT prompting, SFT, and existing chart-specific baselines.