Efficient system recovers text prompts from videos for better controls
Meta Inverse Prompting for Video Generative Models
Computer Vision and Pattern Recognition
Summary
Videos can be made from text or images using computer models, but figuring out what text led to a particular video is much harder. The authors created a new method that can quickly guess the text prompts behind videos by learning across many examples. This helps people better control and edit videos because they can understand what instructions created them. Their method works faster and more accurately than earlier techniques that only worked for images or required slow guessing.
What this means in practice
- •For video production teams: Recover text-based instructions from existing videos to facilitate precise editing and content control.
- •For game developers: Generate editable video assets by inferring prompts from game footage for efficient content updates.
Authors
Yerin Jung, Jongheon Jeong
Abstract
Recent advances in video generative models have enabled flexible conditional generation and editing pipelines, such as text-to-video, image-to-video, and reference-guided synthesis. However, these workflows are inherently one-directional, mapping conditions to videos without providing a mechanism to invert this process. In complex media production settings, the ability to recover controllable textual conditions from a given video is highly desirable, yet remains underexplored. While inverse prompting has been studied for image generation, directly extending these approaches to video is challenging due to the need to capture temporal dynamics beyond static appearance, as well as the substantial computational cost of per-video optimization. In this work, we propose the first practical framework for efficient video inverse prompting. Specifically, we formulate inverse prompting as a meta-learning problem and introduce Meta Video Inverse Prompting (MVIP), which learns a meta-prompt optimized over temporal dimensions to enable fast and effective prompt inference. Our approach amortizes the inversion process, eliminating the need for costly iterative optimization at test time. Experimental results demonstrate that MVIP recovers higher-quality inverse prompts with significantly improved reconstruction fidelity compared to image-based inversion and iterative captioning baselines, while achieving substantial gains in efficiency. Code and video results are available at https://ueriniuno.github.io/mvip-page/.