Interactive video storytelling improves by linking viewer intent and context
VIS-Ground: Video Interactive Storytelling with Contextual Grounding
Computer Vision and Pattern Recognition
Summary
Video interactive storytelling lets viewers influence how a story unfolds in a video. The challenge is that what a viewer wants next depends on both the story source and what’s already happened in the video, but these connections aren’t always obvious. The authors propose VIS-Ground, a system that turns all these mixed hints into clear instructions for video generation. This approach helps make videos that better follow the story and viewer requests, performing better than existing methods in tests.
What this means in practice
- •For interactive media developers: Create interactive video experiences that better respond to viewer requests by grounding content generation in story context and prior video states.
- •For video game designers: Build dynamic narrative-driven game cutscenes adapting to player choices with grounded video generation supporting evolving story paths.
Authors
Bingxuan Li, Yiwen Song, Xueqing Wu, Yanzhou Pan, Yang Li, Kuang Su, Jingyun Liu, Sebastian Ko, Huan Zhang, Tong Zhang, Nanyun Peng, Tomas Pfister, Yale Song
Abstract
Video interactive storytelling enables viewers to actively steer how a video unfolds. However, once we allow viewers to intervene during generation, a new challenge arises: The viewer's request can have latent dependencies on both the grounding source and the current rendered video state. These dependencies may not be explicitly stated in any individual input, but emerge only when the source, rendered history, and new viewer intent are considered jointly. Existing interactive video generation systems primarily emphasize following viewer instructions, while source-grounded video generation methods focus on aligning generated content with an external narrative or knowledge source. This leaves a fundamental question underexplored: What context should a generation model ground on during interactive continuation, and how can heterogeneous, unstructured inputs be transformed into such grounding context? In this work, we formulate contextual grounding as the process of transforming heterogeneous input context into an executable constraint model for video generation. To address this challenge, we introduce VIS-Ground, which performs Structured Context Abstraction to recover grounded states and cross-context dependencies, Generation Constraints Induction to project relevant dependencies into candidate-specific constraints, and Constrained Video Generation to enforce these constraints through planning, verification, revision, and rendering. Across three video generation backbones, VIS-Ground consistently achieves the highest overall composite score, reaching an average absolute improvement of 10.3 points over the strongest per-backbone baselines. Detailed analysis further shows gains across both narrative and knowledge grounding, and reveals remaining challenges in dependency extraction, and faithful realization during video rendering.