Video language models generate intermediate event chains for causal reasoning
From Cause to Effect: Generating Intermediate Event Chains in Videos
Computer Vision and Pattern Recognition
Summary
Video-language models can recognize objects and actions in videos but often fail to understand cause-and-effect relationships. The authors introduce a new task where these models generate a step-by-step chain of events connecting a cause to its effect, which requires true understanding of the story. They also create ways to automatically check if the generated chains make sense. Their tests show current models struggle, but fine-tuning with special rewards helps improve how well these models understand and describe causal sequences in videos.
What this means in practice
- •For video content platforms: Improve automatic video description systems by generating accurate sequences of cause and effect events within video clips.$Commercial implications: Provides technology for video platforms to enhance content understanding and user engagement with detailed causal event narratives.
- •For film editors and storytellers: Assist with identifying and summarizing key causal event chains to analyze narrative structure in short movies and clips.
Authors
Ridouane Ghermi, Vicky Kalogeiton, Ivan Laptev
Abstract
Video-Language Models (VLMs) excel at visual recognition but struggle with causal reasoning. Current evaluations are flawed: multiple-choice QA allows shortcuts, and open-ended QA lacks robust automatic evaluation. We introduce Causal Chain Generation (CCG), a novel task where models have to generate the sequence of intermediate events connecting a cause and an effect. This structured setup demands genuine reasoning and enables reliable automatic scoring. We accompany this task with two complementary metrics: SODA-Embedding, which measures the semantic matching of a generated chain against a reference, and Causal-QA, which probes the chain's causal coherence by asking whether it contains the specific causal facts needed to reconstruct the narrative. We provide two test sets: SF20K-CC, a manually annotated diagnostic on 20 short films, and CMD-CC, a movie clip benchmark derived from audio descriptions. Our analysis reveals that state-of-the-art VLMs struggle with narrative causality understanding, frequently hallucinating or omitting key events. To address this, we fine-tune VLMs with GRPO using a composite reward that combines a format reward, SODA-Embedding, and Causal-QA accuracy, demonstrating substantial improvements over strong baselines.