Assemble improves video question answering with explicit evidence tracing
ASSEMBLE: Atomic Skills for Evidence-Grounded Video Reasoning
Multimedia
Summary
Answering questions about long videos is hard because the right information is spread out across many parts of the video. The authors created ASSEMBLE, a method that tracks and shows exactly which clips and moments support the answers it gives. This makes it easier to check if the reasoning is based on the correct video parts. Their approach also improves accuracy on several video reasoning tests without losing performance compared to other large models.
What this means in practice
- •For video platform developers: Enhance video question answering systems by linking answers to specific video evidence for better user trust and transparency.
- •For digital forensics teams: Improve analysis tools by providing verifiable video evidence references supporting findings in long video investigations.
Authors
Xiyang Wu, Zongxia Li, Shengxin Zhang, Zhichao Liu, Dinesh Manocha
Abstract
Complex video reasoning often depends on evidence scattered across distant moments, entities, and events, yet a correct answer alone does not reveal whether a model relied on the right parts of the video. We introduce ASSEMBLE, a framework that makes supporting evidence explicit throughout long-video reasoning. ASSEMBLE organizes local observations and cross-clip narratives into timestamped evidence catalogs traceable to the source video. A grounding-aware reader then composes question-specific atomic skills whose structured outputs contain explicit evidence references and support assessments. We use correctness-gated citation alignment as a direct grounding signal: after teacher-supervised fine-tuning, Group Relative Policy Optimization (GRPO) jointly optimizes answer correctness and citation alignment. This produces inspectable intermediate traces while keeping final predictions linked to explicit supporting evidence. Using a 9B reader supervised by a 235B teacher and shared precomputed evidence catalogs, ASSEMBLE achieves 59.2% macro-averaged answer accuracy across three long-video reasoning benchmarks, compared with 58.3% for Gemini-2.5-Pro, while improving macro-averaged overlap-based Grounded accuracy by 6.7%, with gains on all three benchmarks. Ablations further show that, with the same post-trained reader and inference budget, structured skill inference improves Grounded accuracy over free-form reasoning. Together, these results show that explicit evidence grounding can be integrated directly into long-video reasoning without sacrificing answer accuracy.