Video question answering challenges revealed by ultra long tv dramas
DramaVQA: Probing Detailed Understanding of Ultra-Long TV Dramas
Computer Vision and Pattern Recognition
Summary
Understanding really long videos, like TV dramas that last many hours, is hard for current AI systems. The authors created a huge new dataset called DramaVQA, with lots of questions about these long dramas to test AI understanding more deeply. They used clever language models to help make the questions and humans to check them for quality. Results show that even the best AI tools today struggle with the hardest questions that require watching the entire show. This means future AI needs better long-term memory and reasoning to understand long videos.
What this means in practice
- •For video platform developers: Test AI tools on ultra-long videos to improve content indexing and user query answering in streaming services.
- •For media analytics teams: Use the dataset to train or evaluate systems that need to analyze and summarize long narrative video content.
Authors
Xinyue Huo, Pengfei Chen, Jiannan Ge, Kaiwen Duan, Jiacheng Shao, Yunning Cao, Lingxi Xie, Qi Tian
Abstract
Despite recent progress in long-form video understanding, current benchmarks mostly focus on short clips or hour-scale videos. This leaves a major gap in evaluating ultra-long video understanding, where videos often last tens of hours. To bridge this gap, we introduce DramaVQA, a large-scale benchmark built upon long-form TV dramas. Containing approximately 20K question-answering (QA) pairs, the scale of DramaVQA exceeds previous long-video datasets, such as Video-MME and LVBench, by an order of magnitude. Our benchmark has two main features that make it stand out from existing datasets. First, we establish a scalable, LLM-assisted annotation pipeline that uses multimodal large language models (MLLMs) to generate complex questions, followed by strict human checking to ensure high data quality. Second, DramaVQA introduces a multi-level time structure; queries are designed across different video lengths, with the hardest questions requiring overall understanding across the entire drama. Extensive evaluations show that state-of-the-art MLLMs face major challenges when tested on these very long videos, highlighting the need for agentic AI algorithms capable of managing long-term memory. The project is available at https://github.com/xinyuehuo/DramaVQA