DataVista evaluates multimodal AI on understanding data videos
DataVista: Diagnosing Multimodal LLMs on Data Video Understanding
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Videos that show changing charts and tell stories with data are common in news and business, but computers find them hard to fully understand. The authors created DataVista, a set of nearly a thousand real data videos with thousands of questions, to test how well AI models can read the charts, connect information over time, and grasp the story told. They tested 19 popular multimodal AI models and found they still struggle, especially with understanding causes and the overall narrative. This new benchmark helps identify where AI needs improvement to better interpret data-driven videos.
What this means in practice
- •For business analysts: Evaluate AI tools for analyzing data-driven videos in reports to improve automated video understanding accuracy.
- •For media production teams: Test and enhance multimodal AI systems that assist in generating or summarizing analytic news videos using DataVista.
Authors
Yupeng Xie, Zhenyang Wang, Jiayi Zhu, Yinghao Tang, Zhouan Shen, Yiyu Chen, Yuyu Luo
Abstract
Data video is a media form that integrates data visualization with video narrative, widely adopted in news reporting and business analysis. Compared with general video understanding, data video understanding places greater emphasis on accurately reading data from animated charts, integrating evidence across charts and time, and understanding how narrative organization and visual design communicate information. Yet existing benchmarks target either general videos or static charts, and data video understanding has not been systematically evaluated. We present DataVista, the first benchmark for data video understanding, containing 961 real-world data videos and 6,775 evaluation questions organized under a three-level progressive capability framework (data perception, temporal reasoning, narrative understanding) with 10 fine-grained question types across five topic domains. Systematic evaluation of 19 mainstream MLLMs shows that the best-performing model, Gemini-3.1-Pro, achieves 70.0% overall accuracy, still far below human expert performance, with models performing worst on Causal Reasoning and Narrative Structure. Increasing frame counts and adding subtitles mainly benefit data perception and temporal reasoning, with limited gains in narrative understanding. Further analysis of model responses identifies typical failure modes in chart reading, evidence judgment, and instruction understanding. The benchmark is available at https://github.com/HKUSTDial/DataVista.