Video editing improved by reasoning powered large language models
ThinkV2V: Unleashing the Reasoning Capability of MLLMs for Instruction-Guided Video Editing
Computer Vision and Pattern Recognition
Summary
Editing videos based on instructions can be tricky when the task involves understanding hidden meanings or cause-effect relationships. The authors propose ThinkV2V, a system that makes large language models think more carefully about the video and the instructions before editing. They train and test the system on a new large dataset designed for challenges needing reasoning. This approach helps their model outperform other larger models on both simple and complex video editing tasks.
What this means in practice
- •For video production teams: Enhance automated video editing tools to handle complex, implicit editing instructions requiring causal and semantic understanding.
- •For mobile app developers: Integrate improved reasoning capabilities into mobile video editing apps to better interpret user instructions and produce desired edits.
Authors
Donghao Zhou, Haoyang He, Fan Zhang, Hao Yang, Guisheng Liu, Xin Gao, Zhongwei Wan, Xingyuan Bu, Jie Wang, Qiangpeng Yang, Shilei Wen, Chi-Wing Fu, Pheng-Ann Heng
Abstract
Instruction-guided video editing has made significant progress, yet existing methods use multimodal large language models (MLLMs) primarily as semantic encoders, so they often fall short in working with implicit edits that require causal or semantic reasoning. To bridge this fundamental gap in video editing, we propose ThinkV2V, a reasoning-driven framework for complex instruction-guided video editing, explicitly activating MLLM thinking before visual generation. At its core, ThinkV2V builds on a practical MLLM-to-DiT architecture to turn explicit thinking over the source video and instruction into refined conditioning signals for video editing. Further, we equip it with a dedicated training and inference recipe, combining Progressive Curriculum Training, which gradually cultivates the model from basic editing to reasoning-intensive cases, with Inference-Time Thinking Scaling, which iteratively refines candidate prompts and selects the most reliable one, to better elicit reasoning in challenging editing scenarios. We also curate the ThinkV2V-150K dataset and introduce ThinkV2V-Bench to support training and evaluation of video editing with implicit intent and causal reasoning. Experimental results demonstrate the state-of-the-art performance of ThinkV2V on both complex and standard editing scenarios, in which our 5B-scale DiT model substantially outperforms larger 10B-scale baselines.