Factor-aware learning improves cinematic video generation from sparse inputs
Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation
Computer Vision and Pattern Recognition
Summary
Creating cinematic videos from just a few cues, like sparse frames or text, is hard because it needs understanding of timing and control over factors such as characters and camera angles. The authors propose a new training approach called Factor-Aware Predictive Learning (FAPL) that helps video generators better use language model features to produce videos with precise control over these factors and predictable timing. Their method trains the video generator and language model together in stages to improve video editing, multi-shot sequences, and sparse input video creation. Experiments show that this approach works better than previous methods.
What this means in practice
- •For video production teams: Generate cinematic videos from sparse inputs with enhanced control over characters, scenes, and camera movements.$Commercial implications: This method enables advanced video editing and generation tools for filmmakers and content creators, improving quality and efficiency.
- •For game developers: Create realistic cinematic sequences for games using limited input cues with factor-aware temporal and content control.
Authors
Songlin Yang, Zhe Wang, Xiaotong Zhao, Xi Wang, Ran Zhang, Shuai Li, Taiyi Wu, Songlian Li, Eric Liu, Alan Zhao, Anyi Rao
Abstract
Cinematic video generation requires both temporal inference from limited inputs (i.e., sparse rather than frame-by-frame guidance) and factor-aware control of character, scene, action, and camera. Multimodal large language models (MLLMs) offer useful priors for both capabilities, but their representations are not automatically actionable for video generation: a Video Diffusion Transformer (DiT) may bypass them through native conditioning pathways, and using them does not ensure either factor-aware control or generation-compatible temporal prediction. We propose Factor-Aware Predictive Learning (FAPL), which establishes effective MLLM latent use as a shared prerequisite and then targets the two capabilities with complementary learning signals. First, latent-only generator grounding trains the DiT with a frozen MLLM and connector, strengthening the DiT's use of MLLM latents for generation. Second, for factor-aware control, paired videos specify a target-factor change and non-target preservation, and factor-span replacement trains the DiT to realize this relationship in its output. Third, for temporal inference, text-only and text-plus-complete-video observations share a video target, and the trained DiT is then frozen while the MLLM and connector are optimized through it. This final stage trains limited-input representations for generation while retaining complete-observation reconstruction as a complementary training mode. Extensive experiments demonstrate our consistently better performance on three cinematic video generation tasks (i.e., sparse-to-video, factor-aware editing, and multi-shot generation) over baselines and validate FAPL's contributions to MLLM feature use, temporal control, and factor coordination.