Multimodal agents struggle to fix and check flawed data visualizations

VisAudit: Evaluating Multimodal Agents for Visual Diagnosis and Repair

Machine Learning

Summary

Charts and graphs often contain mistakes, but current AI tools find it hard to detect and fix these errors on their own. The authors created VisAudit, a test system that gives AI agents flawed charts along with related data, then checks if they can spot and repair the problems correctly without breaking other parts of the visualization. They found that even the best AI model fixed less than half of the flawed charts entirely on its own. This shows there is still a big gap before AI can reliably review and repair visualizations like humans can.

What this means in practice

  • For data visualization engineers: Evaluate and improve AI tools that autonomously detect and fix chart errors using VisAudit’s diverse faulty chart dataset and tasks.
  • For product teams for business intelligence software: Incorporate VisAudit benchmarks to test and enhance automated visualization debugging features in BI tools before launch.$Commercial implications: Enables development and sale of smarter BI platforms with AI-powered visualization error detection and correction capabilities.

Authors

Shicheng Liu, Adam Kahirov, Qi Zhang, Zhimin Hu, Song Wang, Junhong Lin, Julian Shun, Yada Zhu

Abstract

Multimodal agents are increasingly used for data visualization tasks but remain limited in autonomous review. Unlike humans, they may fail to recognize when a visualization is incorrect, determine what to change, repair it without disrupting correct content, and verify whether the intervention succeeded. Existing benchmarks largely evaluate predefined individual capabilities such as chart generation, instruction-guided editing, or defect detection, and therefore do not capture this gap in autonomous review. We introduce VisAudit, a benchmark for evaluating visualization diagnosis, repair, and verification. Given a rendered chart and configurable auxiliary evidence, including its source data table, intended text summary, and visualization code, an agent iteratively diagnoses potential defects, modifies and executes visualization code, inspects execution and visual feedback, and determines when no further intervention is needed. VisAudit defines three tracks spanning diagnosed repair, autonomous repair, and open-world verification, and contains 1,900 flawed instances across 21 chart types and 10 flaw categories, together with 300 initially correct charts. We construct the benchmark through controlled perturbations of validated source visualizations, with systematic verification and human-aligned quality control to ensure that injected defects are well-defined and recoverable from the available evidence. Experiments with leading multimodal models reveal a substantial gap from reliable autonomous review: the strongest evaluated model fully recovers only $47.4\%$ of flawed charts in the autonomous-repair setting.