Long form questions test how well ai understands hours of video
LF-VideoQA: Evaluating Long-Video Understanding with Long-Form Theory-of-Mind Questions
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
Summary
Understanding very long videos is hard because it requires putting together many parts that happen over a long time. The authors made a new set of test questions called LF-VideoQA that focus on what characters know and believe throughout a movie, needing the AI to remember and reason across many scenes. They automatically created this tough dataset using movie scripts and special AI tools, then checked quality with humans. Current AI video models find these questions very challenging and often miss important details.
What this means in practice
- •For video platform developers: Build AI systems that answer complex questions requiring reasoning across hours of video content to improve user engagement features.
- •For movie analytics teams: Automate analysis of character beliefs and plot understanding in films to support content summarization or recommendation systems.
Authors
Shaden Shaar, Bradon Thymes, Majd Aldaye, Claire Cardie, Bharath Hariharan
Abstract
The hardest part of understanding a long video is not pinpointing any one moment but synthesizing many, e.g., answering a question that requires tracking what develops across hours of content and articulating how the pieces fit together. To date, most long-video QA datasets require only a short answer or a multiple-choice selection, oversimplifying the long-video understanding task to retrieval. We introduce LF-VideoQA, a dataset for long-form, open-ended question answering over long narrative videos built on theory-of-mind questions: a character's belief depends on everything they have seen or been told, so each question demands assembling and reasoning from evidence distributed across the video. The dataset contains more than 6,300 questions on 94 recent films; each requires reasoning over scenes spanning an average of 23 minutes and comes with a question-specific rubric for recall and precision scoring. The dataset is created entirely automatically by annotating screenplays with scene events and each character's first-order beliefs and meta-beliefs, then generating (question, answer, rubric) triples through LLM-based question generation and verification; human studies validate their quality. We evaluate four families of state-of-the-art open-source VLMs across three settings and find the task highly challenging: (i) even an oracle with access to the screenplay falls well short of complete answers; (ii) the best non-oracle model attains barely half of the recall level of the oracle; (iii) unlike on prior long-video benchmarks, captions and subtitles underperform the video itself, indicating that the questions need visual evidence; and (iv) recall is the predominant bottleneck, with answers omitting many required rubric claims.